Too Many Secrets
A wise man once said, “you shouldn’t use ENV variables for secret
In large part, he was right, for all the reasons he gives (and you should read
them). Filesystem locations are usually a better operating system interface to
communicate secrets than environment variables; fewer things can intercept an
open() than can read your process’s command-line or calling environment.
One might say that files are “more secure” than environment variables. To his credit, Diogo doesn’t, for good reason: one shouldn’t refer to the superiority of such a mechanism as being “more secure” in general, but rather, as better for a specific reason in some specific circumstance.
Supplying your PyPI password to tools you run on your personal machine is a very different case than providing a cryptographic key to a containerized application in a remote datacenter. In this case, based on the constraints of the software presently available, I believe an environment variable provides better security, if you use it correctly.
Popping A Shell By Any Other Name
If you upload packages to the python package index, and
people use those packages, your PyPI password is an extremely high-privilege
credential: effectively, it grants a time-delayed arbitrary code execution
privilege on all of the systems where anyone might
pip install your packages.
Unfortunately, the suggested mechanism to manage this crucial, potentially world-destroying credential is to just stick it in an unencrypted file.
The authors of this documentation know this is a problem; the authors of the tooling know too (and, given that these tools are all open source and we all could have fixed them to be better about this, we should all feel bad).
Leaving the secret lying around on the filesystem is a form of ambient authority; a permission you always have, but only sometimes want. One of the worst things about this is that you can easily forget it’s there if you don’t use these credentials very often.
The keyring is a much better place, but even it can be a slightly scary place to put such a thing, because it’s still easy to put it into a state where some random command could upload a PyPI release without prompting you. PyPI is forever, so we want to measure twice and cut once.
Luckily, even more secure places exist: password managers. If you use
https://1password.com or https://www.lastpass.com, both offer command-line
interfaces that integrate nicely with PyPI. If you use 1password, you’ll
really want https://stedolan.github.io/jq/ (
apt-get install jq,
jq) to slice & dice its command-line.
The way that I manage my PyPI credentials is that I never put them on my filesystem, or even into my keyring; instead, I leave them in my password manager, and very briefly toss them into the tools that need them via an environment variable.
First, I have the following shell function, to prevent any mistakes:
1 2 3 4
1 2 3 4 5 6
This way I can debug Twine, my
setup.py, and various test-upload things
without ever needing real credentials at all.
But, OK. Eventually, I need to actually get the credentials and do the thing. How does that work?
1password’s command line is a little tricky to log in to (you have to
its output, it’s not just a command), so here’s a handy shell function that
will do it.
1 2 3 4 5 6
Then, I have this little helper for slicing out a particular field from the OP JSON structure:
1 2 3
And finally, I use this to grab the item I want (named, memorably enough, “PyPI”) and invoke Twine:
1 2 3 4 5 6 7
For lastpass, you can just log in (for all shells; it’s a little less secure)
lpass login; if you’ve logged in before you often don’t even have to do
that, and it will just prompt you when running command that require you to be
logged in; so we don’t need the preamble that 1password’s command line did.
Its version of
prod.twine looks quite similar, but its plaintext output
obviates the need for
1 2 3 4 5
“Keep secrets out of your environment” is generally a good idea, and you should always do it when you can. But, better a moment in your process environment than an eternity on your filesystem. Environment-based configuration can be a very useful stopgap for limiting the lifetimes of credentials when your tools don’t support more sophisticated approaches to secret storage.1
If you are interested in secure secret storage, my micro-project
secretly might be of interest. Right
now it doesn’t do a whole lot; it’s just a small wrapper around the excellent
keyring module and the
pinentry-mac password prompt tools.
secretly presents an interface both for prompting users for their credentials
without requiring the command-line or env vars, and for saving them away in
keychain, for tools that need to pull in an API key and don’t want to make
the user manually edit a config file first.
Really, PyPI should have API keys that last for some short amount of time, that automatically expire so you don’t have to freak out if you gave somebody a 5-year-old laptop and forgot to wipe it first. But again, if I wanted that so bad, I should have implemented it myself... ↩