Unproblematize

I’ve got 999 problems.

The essence of software engineering is solving problems.

The first impression of this insight will almost certainly be that it seems like a good thing. If you have a problem, then solving it is great!

But software engineers are more likely to have mental health problems1 than those who perform mechanical labor, and I think our problem-oriented world-view has something to do with that.

So, how could solving problems be a problem?


As an example, let’s consider the idea of a bug tracker.

For many years, in the field of software, any system used to track work has been commonly referred to as a “bug tracker”. In recent years, the labels have become more euphemistic and general, and we might now call them “issue trackers”. We have Sapir-Whorfed2 our way into the default assumption that any work that might need performing is a degenerate case of a problem.

We can contrast this with other fields. Any industry will need to track work that must be done. For example, in doing some light research for this post, I discovered that the relevant term of art in construction3 is typically “Project Management” or “Task Management” software. “Projects” and “Tasks” are no less hard work, but the terms do have a different valence than “Bugs” and “Issues”.

I don’t think we can start to fix this ... problem ... by attempting to change the terminology. Firstly, the domain inherently lends itself to this sort of language, which is why it emerged in the first place.

Secondly, Atlassian has desperately been trying to get everybody to call their bug tracker a “software development tool” where you write “stories” for years, and nobody does. It’s an issue tracker where you file bugs, and that’s what everyone calls it and describes what they do with it. Even they have to protest, perhaps a bit too much, that it’s “way more than a bug and issue tracker”4.


This pervasive orientation towards “problems” as the atom of work does extend to any knowledge work, and thereby to any “productivity system”. Any to-do list is, at its core, a list of problems. You wouldn’t put an item on the list if you were happy with the way the world was. Therefore every unfinished item in any to-do list is a little pebble of worry.

As of this writing, I have almost 1000 unfinished tasks on my personal to-do list.

This is to say nothing of any tasks I have to perform at work, not to mention the implicit א‎0 of additional unfinished tasks once one considers open source issue trackers for projects I work on.

It’s not really reasonable to opt out of this habit of problematizing everything. This monument to human folly that I’ve meticulously constructed out of the records of aspirations which exceed my capacity is, in fact, also an excellent prioritization tool. If you’re a good engineer, or even just good at making to-do lists, you’ll inevitably make huge lists of problems. On some level, this is what it means to set an intention to make the world — or at least your world — better.

On a different level though, this is how you set out to systematically give yourself anxiety, depression, or both. It’s clear from a wealth of neurological research that repeated experiences and thoughts change neural structures5. Thinking the same thought over and over literally re-wires your brain. Thinking the thought “here is another problem” over and over again forever is bound to cause some problems of its own.

The structure of to-do apps, bug trackers and the like is such that when an item is completed — when a problem is solved — it is subsequently removed from both physical view and our mind’s eye. What would be the point of simply lingering on a completed task? All the useful work is, after all, problems that haven’t been solved yet. Therefore the vast majority of our time is spent contemplating nothing but problems, prompting the continuous potentiation6 of neural pathways which lead to despair.


I don’t want to pretend that I have a cure for this self-inflicted ailment. I do, however, have a humble suggestion for one way to push back just a little bit against the relentless, unending tide of problems slowly eroding the shores of our souls: a positivity journal.

By “journal”, I do mean a private journal. Public expressions of positivity7 can help; indeed, some social and cultural support for expressing positivity is an important tool for maintaining a positive mind-set. However, it may not be the best starting point.

Unfortunately, any public expression becomes a discourse, and any discourse inevitably becomes a dialectic. Any expression of a view in public is seen by some as an invitation to express its opposite8. Therefore one either becomes invested in defending the boundaries of a positive community space — a psychically exhausting task in its own right — or one must constantly entertain the possibility that things are, in fact, bad, when one is trying to condition one’s brain to maintain the ability to recognize when things are actually good.

Thus my suggestion to write something for yourself, and only for yourself.

Personally, I use a template that I fill out every day, with four sections:

  • “Summary”. Summarize the day in one sentence that encapsulates its positive vibes. Honestly I put this in there because the Notes app (which is what I’m using to maintain this) shows a little summary of the contents of the note, and I was getting annoyed by just seeing “Proud:” as the sole content of that summary. But once I did so, I found that it helps to try to synthesize a positive narrative, as your brain may be constantly trying to assemble a negative one. It can help to write this last, even if it’s up at the top of your note, once you’ve already filled out some of the following sections.

  • “I’m proud of:”. First, focus on what you personally have achieved through your skill and hard work. This can be very difficult, if you are someone who has a habit of putting yourself down. Force yourself to acknowledge that you did something useful, even if you didn’t finish anything, you almost certainly made progress and that progress deserves celebration.

  • “I’m grateful to:”. Who are you grateful to? Why? What did they do for you? Once you’ve made the habit of allowing yourself to acknowledge your own accomplishments, it’s easy to see those; pay attention to the ways in which others support and help you. Thank them by name.

  • “I’m lucky because:”. Particularly in post-2020 hell-world it’s easy to feel like every random happenstance is an aggravating tragedy. But good things happen randomly all the time, and it’s easy to fail to notice them. Take a moment to notice things that went well for no good reason, because you’re definitely going to feel attacked by the universe when bad things happen for no good reason; and they will.

Although such a journal is private, it’s helpful to actually write out the answers, to focus on them, to force yourself to get really specific.

I hope this tool is useful to someone out there. It’s not going to solve any problems, but perhaps it will make the world seem just a little brighter.


  1. “Maintaining Mental health on Software Development Teams”, Lena Kozar and Vova Vovk, in InfoQ 

  2. Wikipedia page for “Linguistic Relativity” 

  3. “Construction Task and Project Tracking”, from Raptor Project Management Software 

  4. Jira Features List, Atlassian Software 

  5. “Culture Wires the Brain: A Cognitive Neuroscience Perspective”, Denise C. Park and Chih-Mao Huang, Perspect Psychol Sci. 2010 Jul 1; 5(4): 391–400. 

  6. Long-term potentiation and learning, J L Martinez Jr, B E Derrick 

  7. The #PositivePython hashtag on Twitter was a lovely experiment and despite my cautions here about public solutions to this problem, it’s generally pleasant to participate in. 

  8. As we well know. 

Announcing Pomodouroboros

I wrote my own pomodoro timer which is also a meditation on mortality.

As I mentioned previously, I’ve recently been medicated for ADHD.

Everyone’s experience with medication, even the same medication, is different, but my particular experience — while hugely positive — has involved not so much a decrease in symptoms, but rather a shifting of my symptom profile. Some of my executive functions (particularly task initiation) have significantly improved, but other symptoms, such as time blindness have gotten significantly worse. This means, for example, I can now easily decide to perform a task, and actually maintain focus on that task for hours1, but it’s harder to notice that it’s time to stop, and still somewhat difficult to tear myself away from it.

I’ve tried pomodoro timers before and I’ve had mixed success with them. While I tend to get more done if I set a pomodoro, it’s hard to remember to set the timers in the first place, and it’s hard to do the requisite time-keeping to remember how many pomodoros I’ve already set, how many more I’ll have the opportunity to set, etc. Physical timers have no associated automation and data recording, and apps can be so unobtrusive that I can easily forget about them entirely. I’ve long had an aspiration to eventually write my own custom-tailored app that addresses some of these issues.

As part of a renewed interest in ADHD management techniques, I watched this video about ADHD treatments from Dr. Russell Barkley, wherein he said (I’m paraphrasing) “if I don’t put an intervention into your visual field it might as well not exist”.

I imagined timer that:

  1. was always clearly present in my visual field;
  2. recorded the passage of intervals of time regardless of any active engagement from the user; the idea is to record the progress of the day, not give you a button you need to remember to push;
  3. rewarded me for setting active intentions about what to do with those chunks of time, and allowed me to mark them as successful or failed.

So, last weekend, leveraging my newly enhanced task-initiation and concentration-maintenance abilities, I wrote it, and I’ve been using it all week. Introducing Pomodouroboros, the pomodoro timer that reminds you that the uncaring void marches on regardless of your plans or intentions.

I’ve been using it all week and preliminary results are extremely positive.

This thing is in an extremely rough state. It has no tests, no docs, and an extremely inscrutable UI that you need to memorize in order to use effectively. I need plenty of help with it. I contemplated keeping it private and just shipping a binary, but a realistic assessment of my limited time resources forced me to admit that it already kind of does what I need, and if I want to enhance it to the point where it can help other people, I’ll need plenty of help.

If this idea resonates with you, and you’re on macOS, check out the repo, make a virtualenv somehow, install its dependencies, I don’t know how you make virtualenvs or install dependencies, I’m not your dad2, and run ./runme. If you’re on another platform, check out the code, ask me some questions, and maybe try to write a port to one of them.


  1. I cannot express how alien the sensation is to have conscious control over initiating this process; I’ve certainly experienced hyperfocus before but it’s always been something that happens to me and not something that I do 

  2. If I am your dad, come talk to me, based on your family history it’s quite likely that you do have ADHD and I’m happy to talk about how to get this installed for you offline. 

Interfaces and Protocols

Comparing zope.interface and typing.Protocol.

Some of you read my previous post on typing.Protocols and probably wondered: “what about zope.interface?” I’ve advocated strongly for it in the past — but now that we have Mypy and Protocols, is it simply a relic of an earlier time? Can we entirely replace it with Protocol?

Let’s have a look.

Typing in 2 dimensions

In the previous post I discussed structural versus nominal typing. In Mypy’s type system, most classes are checked nominally whereas Protocol is checked structurally. However, there’s another way that Protocol is distinct from a normal class: normal classes are concrete types, and Protocols are abstract.

Abstract types:

  1. cannot be instantiated: every instance of an abstract type is an instance of some concrete sub-type, and
  2. do not include (complete) implementation logic.

Concrete types:

  1. can be instantiated: they are complete descriptions of a type, and
  2. must include all their own implementation logic.

Protocols and Interfaces are both abstract, but Interfaces are nominal. The highest level distinction between the two is that when you have a problem that requires an abstract type, but nominal checking is preferable to structural, Interfaces are a better solution.

Python’s built-in Abstract Base Classes are technically abstract-and-nominal as well, but they’re in a strange halfway space; they’re formally “abstract” because they can’t be instantiated, but they’re partially concrete in that they can contain any amount of implementation logic themselves, and thereby making an object which is a subtype of multiple ABCs drags in all the usual problems of the conflicting namespaces within multiple inheritance.

Theoretically, there’s a way to treat ABCs as purely abstract — which is to use ABCMeta.register — but as of this writing (March 2021) it doesn’t work with Mypy, so within the context of “static typing in Python” we presently have to ignore it.

Practicalities

The first major advantage that Protocol has is that since it is now built in to Python itself, there’s no reason not to use it. When Protocol didn’t even exist, regardless of all the advantages of adding explicit abstract types to your project with zope.interface, it did still have the small down-side of requiring a new dependency, with all the minor headaches that might imply.

beyond the theoretical distinctions, there’s a question of how well tooling supports zope.interface. There are some clear gaps; there is not a ton of great built-in IDE support for zope.interface; less-sophisticated linters will sometimes still complain that Interfaces don’t take self as their first argument. Indeed, Mypy itself does this by default — although more on that in a moment. Less mainstream performance-focused type-checkers like Pyre and Pyright don’t support zope.interface, either, although their lack of support for zope.interface is just a part of a broader problem of their lack of extensibility; they also can’t support SQLAlchemy or the Django ORM without special-casing in the tools themselves.

But what about Mypy itself — if we have to discount ABCMeta.register due to practical tooling deficiencies even if they provide a built-in way to declare a nominal-but-abstract type in principle, we need to be able to use zope.interface within Mypy as well for a fair comparison with Protocol. Can we?

Luckily, yes! Thanks to Shoobx, there’s a fairly actively maintained Mypy plugin that supports zope.interface which you can use to statically check your Interfaces.

However, this plugin does have a few key limitations as of this writing (Again, March 2021), which makes its safety guarantees a bit lower-quality than Protocol.

The net result of this is that Protocols have the “home-field advantage” in most cases; out of the box, they’ll work more smoothly with your existing editor / linter setup, and as long as your project supports Python 3.6+, at worst (if you can’t use Python 3.7, where Protocol is built in to typing) you have to take a type-check-time dependency on the typing_extensions package, whereas with zope.interface you’ll need both the run-time dependency of zope.interface itself and the Mypy plugin at type-checking time.

So in a situation where both are roughly equivalent, Protocol tends to win by default. There are undeniably big areas where Interfaces and Protocols overlap, and in plenty of them, using Protocol is a fine idea. But there are still some clear places that zope.interface shines.

First, let’s look at a case which Interfaces handle more gracefully than Protocols: opting out of matching a simple shape, where the shape doesn’t fully describe its own meaning.

Where Interfaces work best: hidden and complex meanings

The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

Alan Perlis, “Epigrams in Programming”, Epigram 34.

The place where structural typing has the biggest advantage is when the type system is expressive enough to fully encode the meaning of the desired behavior within the structure of the type itself. Consider a Protocol which describes an object that can add some integers together:

1
2
3
class Math(Protocol):
    def add_integers(addend1: int, addend2: int) -> int:
        ...

It’s fairly unambiguous what adherents to this Protocol should do, and anyone implementing such a thing should be able to clearly tell that the method is supposed to add a couple of integers together; there’s nothing hidden about the structure of the integers, no constraints the type system won’t let us specify. It would be quite surprising if anything that didn’t have the intended behavior would match this Protocol.

A the other end of the spectrum, we might have a plugin Interface that has a lot of hidden structure. For this example, we have an Interface called IPlugin containing a method with an easy-to-conflict-with name (“name”) overloaded with very specific constraints on its return type: the string must contain the dotted-path name of a Python object in an import-able module (like, for example, "os.path.join").

1
2
3
class IPlugin(Interface):
    def name() -> str:
        "Return the fully-qualified Python identifier of the thing to load."

With Protocols, you can work around these limitations, by manually making it harder to match; adding elements to the structure that embed names relevant to its semantics and thereby making the type behave more as if it were nominally typed.

You could make the method’s name long and ugly instead (plugin_name_to_load, let’s say) or add unused additional attributes (yep_i_am_a_plugin = Literal[True]) in order to reduce the risk of accidental matches, but these workarounds look hacky, and they have to be manually namespaced; if you want to mark it as having semantics associated with your specific plugin system, you have to embed the name of that system in your attributes themselves; here we’re just saying “plugin” but if we want to be truly careful, we have to embed the whole name of our project in there.

With Interfaces, the maintainer of each implementation must explicitly opt in, by choosing whether to specify that they are an @implementer(IPlugin). Since they had to import IPlugin from somewhere, this annotation carries with it a specific, namespaced declaration of semantic intent: “I know what the Interface IPlugin means, and I promise that I can provide it”.

This is the most salient distinction between Protocols and Interfaces: if you have strong reasons to want adherents to the abstract type to opt in, you want an Interface; if you want them to match automatically, you want a Protocol.

Runtime support

Interfaces also provide a more nuanced set of runtime checks.

You can say that an object directlyProvides an interface, allowing for some level of (at least runtime) type safety, and ask if IPlugin is .providedBy some object.

You can do most of this with Protocol, but it’s awkward. The @runtime_checkable decorator allows your Protocol to make isinstance(x, MyProtocol) work like IMyInterface.providedBy(x), but:

  1. you’re still missing directlyProvides; the runtime checking is all by type, not by the individual properties of the instance;
  2. it’s not the default, so if you’re not the one defining the Protocol, there’s no guarantee you’ll be able to use it.

With Interfaces, there’s also no mandatory relationship between the implementer (i.e. the type whose instances fit the specified shape) and the provider (the specific object which can fit the specified shape). This means you get features like classProvides and moduleProvides “for free”.

Interfaces work particularly well for communication between frameworks and application code. For example, let’s say you’re evolving the meaning of an Interface implemented by applications over time — EventHandler, EventHandler2, EventHandler3 — which have similarly named and typed methods, but subtly different expectations on their lifecycle or when precisely the methods will be called. A framework facing this problem can use a series of Interfaces, and check at runtime to see which of these the application implements, and be secure in the knowledge that the application has properly intentionally adopted the new interface, and doesn’t just happen to have a matching method name against an older version.

Finally, zope.interface gives you adaptation and adapter registries, which can be a useful mechanism for doing things like templating, like a much more powerful version of singledispatch from the standard library.

Adapter registries are nuanced, complex tools and unfortunately an example that captures the full utility of their power would itself be commensurately complex. However, the core of adaptation is the idea that if you have an arbitrary object x, and you want a provider of the interface IY, you can do the following:

1
y = IY(x, None)

This performs a multi-stage check:

  1. If x already provides IY (either via implementer, provider, directlyProvides, classProvides, or moduleProvides), it’s simply returned; so you don’t need to special-case the case where you’ve already got what you want.
  2. If x has a __conform__(interface) method, it’ll be called with IY as the interface, and if __conform__ returns anything non-None that result will be returned from the call to IY.
  3. If IY has a specially-defined __adapt__ method, it can implement its own logic for this hook directly.
  4. Each globally-registered function in zope.interface’s adapter_hooks will be invoked to find a function that can transform x into an IY provider. Twisted has its own global registry in this list, which is what registerAdapter manipulates.

But from the perspective of the caller, you can just say “I want an IY”.

With Protocols, you can emulate this with functools.singledispatch by making a function which returns your Protocol type and registers various types to do conversion. The place that adapter registries have an advantage is their central nature and consistent idiom for converting to the target type; you can use adaptation for any Interface in the same way, and any type can participate in adaptation in the ways listed above via flexible mechanisms depending on where it makes sense to put your implementation, whereas any singledispatch function to convert to a Protocol needs to be bespoke per-Protocol.

Describing and restricting existing shapes

There are still several scenarios where Protocol’s semantics apply more cleanly.

Unlike Interfaces, Protocols can describe the types of things that already exist. To see when that’s an advantage, consider a sprawling application that uses tons of libraries and manipulates 3D spatial data points.

There’s a convention among these disparate libraries where they all represent a “point” as an object with .x, .y, and .z attributes which are all floats. This is a natural enough shape, given the domain, that lots of your libraries just fit it by accident. You want to write functions that can work with data output by any of these libraries as long as it plausibly looks like your own concept of a Point:

1
2
3
4
class Point(Protocol):
    x: float
    y: float
    z: float

In this case, the thing defining the Protocol is your application; the thing implementing the Protocol is your collection of libraries. Since the libraries don’t and can’t know about the application — the dependency arrow points the other way — they can’t reference the Protocol to note that they implement it.

Using Protocol, you can also restrict an existing type to preserve future flexibility.

For example, let’s say we’re implementing a “mailbox” type pattern, where some systems deliver messages and other systems retrieve them later. To avoid mix-ups, the system that sends the messages shouldn’t retrieve them and vice versa - receivers only receive, and senders only send. With Protocols, we can describe this without having any new custom concrete types, like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from typing import Protocol, TypeVar

T_co = TypeVar("T_co", covariant=True)
T_con = TypeVar("T_con", contravariant=True)

class Sender(Protocol[T_con]):
    def add(self, item: T_con) -> None:
        "Put an item in the slot."

class Receiver(Protocol[T_co]):
    def pop(self) -> T_co:
        "Retrieve an item from the PO box."

All of that code is just telling Mypy our intentions; there’s no behavior here yet.

The actual implementation is even shorter:

1
2
3
from typing import Set

mailbox: Set[int] = set()

Literally no code of our own - set already does the job we described. And how do we use this?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
def send(sender: Sender[int]) -> None:
    sender.add(3)

def receive(receiver: Receiver[int]) -> None:
    receiver.pop()
    receiver.add(3)
    # Mypy stops us making this mistake:
    # "Receiver[int]" has no attribute "add"

send(mailbox)
receive(mailbox)

For its initial implementation, this system requires nothing beyond types available in the standard library; just a set. However, by treating their parameter as a Sender and a Receiver respectively rather than a Set, send and receive prevent themselves from using any functionality from the set passed in aside from the one method that their respective roles are supposed to “see”. As a result, Mypy will now tell us if any code which receives the sender object tries to remove objects.

This allows us to use existing data structures in libraries without the usual attendant problem of advertising to all clients that every tiny implementation detail of those existing structures is an intended part of the public interface. Python has always tried to make these sort of distinctions by leaving certain things undocumented or saying narratively which things you should rely on, but it’s always hit-or-miss (usually miss) whether library consumers will see those admonitions or not; by making it a feature of the programming environment, Mypy makes it harder to ignore.

Conclusions

In modern Python code, when you have an abstract collection of behavior, you should probably consider using a Protocol to describe it by default. However, Interface is also staying up to date with modern Python tooling by with Mypy support, and it can be worthwhile for more sophisticated consumers that want support for nominal typing, or that want to draw on its reach adaptation and component registration feature-set.

Nice Animations with Twisted and PyGame

Flicker-free, time-accurate animation and movement using LoopingCall.

SNEKS

One of my favorite features within Twisted — but also one of the least known — is LoopingCall.withCount, which can be used in applications where you have some real-time thing happening, which needs to keep happening at a smooth rate regardless of any concurrent activity or pauses in the main loop. Originally designed for playing audio samples from a softphone without introducing a desync delay over time, it can also be used to play animations while keeping track of their appropriate frame.

LoopingCall is all around a fun tool to build little game features with. I’ve built a quick little demo to showcase some discoveries I’ve made over a few years of small hobby projects (none of which are ready for an open-source release) over here: DrawSnek.

This little demo responds to 3 key-presses:

  1. q quits. Always a useful thing for full-screen apps which don’t always play nice with C-c :).
  2. s spawns an additional snek. Have fun, make many sneks.
  3. h introduces a random “hiccup” of up to 1 full second so you can see what happens visually when the loop is overburdened or stuck.

Unfortunately a fully-functioning demo is a bit lengthy to go over line by line in a blog post, so I’ll just focus on a couple of important features for stutter- and tearing-resistant animation & drawing with PyGame & Twisted.

For starters, you’ll want to use a very recent prerelease of PyGame 2, which recently added support for vertical sync even without OpenGL mode; then, pass the vsync=1 argument to set_mode:

1
2
3
4
5
screen = pygame.display.set_mode(
    (640 * 2, 480 * 2),
    pygame.locals.SCALED | pygame.locals.FULLSCREEN,
    vsync=1
)

To allow for as much wall-clock time as possible to handle non-drawing work, such as AI and input handling, I also use this trick:

1
2
3
4
5
6
7
def drawScene():
    screen.fill((0, 0, 0))
    for drawable in self.drawables:
        drawable.draw(screen)
    return deferToThread(pygame.display.flip)

LoopingCall(drawScene).start(1 / 62.0)

By deferring pygame.display.flip to a thread1, the main loop can continue processing AI timers, animation, network input, and user input while blocking and waiting for the vertical blank. Since the time-to-vblank can easily be up to 1/120th of a second, this is a significant amount of time! We know that the draw won’t overlap with flip, because LoopingCall respects Deferreds returned from its callable and won’t re-invoke you until the Deferred fires.

Drawing doesn’t use withCount, because it just needs to repeat about once every refresh interval (on most displays, about 1/60th of a second); the vblank timing is what makes sure it lines up.

However, animation looks like this:

1
2
3
def animate(self, frameCount):
    self.index += frameCount
    self.index %= len(self.images)

We move the index forward by however many frames it’s been, then be sure it wraps around by modding it by the number of frames.

Similarly, the core2 of movement looks like this:

1
2
3
def move(self, frameCount):
    self.sprite.x += frameCount * self.dx
    self.sprite.y += frameCount * self.dy

Rather than moving based on the number of times we’ve been called, which can result in slowed-down movement when the framerate isn’t keeping up, we jump forward by however many frames we should have been called at this point in time.

One of these days, maybe I’ll make an actual game, but in the meanwhile I hope you all enjoy playing with these fun little basic techniques for using Twisted in your game engine.


  1. I’m mostly sure that this is safe, but, it’s definitely the dodgiest thing here. If you’re going to do this, make sure that you never do any drawing outside of the draw() method. 

  2. Hand-waving over a ton of tedious logic to change direction before we go out of bounds... 

Never Run ‘python’ In Your Downloads Folder

Python can execute code. Make sure it executes only the code you want it to.

One of the wonderful things about Python is the ease with which you can start writing a script - just drop some code into a .py file, and run python my_file.py. Similarly it’s easy to get started with modularity: split my_file.py into my_app.py and my_lib.py, and you can import my_lib from my_app.py and start organizing your code into modules.

However, the details of the machinery that makes this work have some surprising, and sometimes very security-critical consequences: the more convenient it is for you to execute code from different locations, the more opportunities an attacker has to execute it as well...

Python needs a safe space to load code from

Here are three critical assumptions embedded in Python’s security model:

  1. Every entry on sys.path is assumed to be a secure location from which it is safe to execute arbitrary code.
  2. The directory where the “main script” is located is always on sys.path.
  3. When invoking python directly, the current directory is treated as the “main script” location, even when passing the -c or -m options.

If you’re running a Python application that’s been installed properly on your computer, the only location outside of your Python install or virtualenv that will be automatically added to your sys.path (by default) is the location where the main executable, or script, is installed.

For example, if you have pip installed in /usr/bin, and you run /usr/bin/pip, then only /usr/bin will be added to sys.path by this feature. Anything that can write files to that /usr/bin can already make you, or your system, run stuff, so it’s a pretty safe place. (Consider what would happen if your ls executable got replaced with something nasty.)

However, one emerging convention is to prefer calling /path/to/python -m pip in order to avoid the complexities of setting up $PATH properly, and to avoid dealing with divergent documentation of how scripts are installed on Windows (usually as .exe files these days, rather than .py files).

This is fine — as long as you trust that you’re the only one putting files into the places you can import from — including your working directory.

Your “Downloads” folder isn’t safe

As the category of attacks with the name “DLL Planting” indicates, there are many ways that browsers (and sometimes other software) can be tricked into putting files with arbitrary filenames into the Downloads folder, without user interaction.

Browsers are starting to take this class of vulnerability more seriously, and adding various mitigations to avoid allowing sites to surreptitiously drop files in your downloads folder when you visit them.1

Even with mitigations though, it will be hard to stamp this out entirely: for example, the Content-Disposition HTTP header’s filename* parameter exists entirely to allow the the site to choose the filename that it downloads to.

Composing the attack

You’ve made a habit of python -m pip to install stuff. You download a Python package from a totally trustworthy website that, for whatever reason, has a Python wheel by direct download instead of on PyPI. Maybe it’s internal, maybe it’s a pre-release; whatever. So you download totally-legit-package.whl, and then:

1
2
~$ cd Downloads
~/Downloads$ python -m pip install ./totally-legit-package.whl

This seems like a reasonable thing to do, but unbeknownst to you, two weeks ago, a completely different site you visited had some XSS JavaScript on it that downloaded a pip.py with some malware in it into your downloads folder.

Boom.

Demonstrating it

Here’s a quick demonstration of the attack:

1
2
3
4
5
~$ mkdir attacker_dir
~$ cd attacker_dir
~/attacker_dir$ echo 'print("lol ur pwnt")' > pip.py
~/attacker_dir$ python -m pip install requests
lol ur pwnt

PYTHONPATH surprises

Just a few paragraphs ago, I said:

If you’re running a Python application that’s been installed properly on your computer, the only location outside of your Python install or virtualenv that will be automatically added to your sys.path (by default) is the location where the main executable, or script, is installed.

So what is that parenthetical “by default” doing there? What other directories might be added?

Anything entries on your $PYTHONPATH environment variable. You wouldn’t put your current directory on $PYTHONPATH, would you?

Unfortunately, there’s one common way that you might have done so by accident.

Let’s simulate a “vulnerable” Python application:

1
2
3
4
5
# tool.py
try:
    import optional_extra
except ImportError:
    print("extra not found, that's fine")

Make 2 directories: install_dir and attacker_dir. Drop this in install_dir. Then, cd attacker_dir and put our sophisticated malware there, under the name used by tool.py:

1
2
# optional_extra.py
print("lol ur pwnt")

Finally, let’s run it:

1
2
~/attacker_dir$ python ../install_dir/tool.py
extra not found, that's fine

So far, so good.

But, here’s the common mistake. Most places that still recommend PYTHONPATH recommend adding things to it like so:

1
export PYTHONPATH="/new/useful/stuff:$PYTHONPATH";

Intuitively, this makes sense; if you’re adding project X to your $PYTHONPATH, maybe project Y had already added something, maybe not; you never want to blow it away and replace what other parts of your shell startup might have done with it, especially if you’re writing documentation that lots of different people will use.

But this idiom has a critical flaw: the first time it’s invoked, if $PYTHONPATH was previously either empty or un-set, this then includes an empty string, which resolves to the current directory. Let’s try it:

1
2
3
~/attacker_dir$ export PYTHONPATH="/a/perfectly/safe/place:$PYTHONPATH";
~/attacker_dir$ python ../install_dir/tool.py
lol ur pwnt

Oh no! Well, just to be safe, let’s empty out $PYTHONPATH and try it again:

1
2
3
~/attacker_dir$ export PYTHONPATH="";
~/attacker_dir$ python ../install_dir/tool.py
lol ur pwnt

Still not safe!

What’s happening here is that if PYTHONPATH is empty, that is not the same thing as it being unset. From within Python, this is the difference between os.environ.get("PYTHONPATH") == "" and os.environ.get("PYTHONPATH") == None.

If you want to be sure you’ve cleared $PYTHONPATH from a shell (or somewhere in a shell startup), you need to use the unset command:

1
2
~/attacker_dir$ python ../install_dir/tool.py
extra not found, that's fine

Setting PYTHONPATH used to be the most common way to set up a Python development environment; hopefully it’s mostly fallen out of favor, with virtualenvs serving this need better. If you’ve got an old shell configuration that still sets a $PYTHONPATH that you don’t need any more, this is a good opportunity to go ahead and delete it.

However, if you do need an idiom for “appending to” PYTHONPATH in a shell startup, use this technique:

1
2
export PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}new_entry_1"
export PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}new_entry_2"

In both bash and zsh, this results in

1
2
$ echo "${PYTHONPATH}"
new_entry_1:new_entry_2

with no extra colons or blank entries on your $PYTHONPATH variable now.

Finally: if you’re still using $PYTHONPATH, be sure to always use absolute paths!

There are a bunch of variant unsafe behaviors related to inspecting files in your Downloads folder by doing anything interactive with Python. Other risky activities:

  • Running python ~/Downloads/anything.py (even if anything.py is itself safe) from anywhere - as it will add your downloads folder to sys.path by virtue of anything.py’s location.
  • Jupyter Notebook puts the directory that the notebook is in onto sys.path, just like Python puts the script directory there. So jupyter notebook ~/Downloads/anything.ipynb is just as dangerous as python ~/Downloads/anything.py.

Get those scripts and notebooks out of your downloads folder before you run ’em!

But cd Downloads and then doing anything interactive remains a problem too:

  • Running a python -c command that includes an import statement while in your ~/Downloads folder
  • Running python interactively and importing anything while in your ~/Downloads folder

Remember that ~/Downloads/ isn’t special; it’s just one place where unexpected files with attacker-chosen filenames might sneak in. Be on the lookout for other locations where this is true. For example, if you’re administering a server where the public can upload files, make extra sure that neither your application nor any administrator who might run python ever does cd public_uploads.

Maybe consider changing the code that handles uploads to mangle file names to put a .uploaded at the end, avoiding the risk of a .py file getting uploaded and executed accidentally.

Mitigations

If you have tools written in Python that you want to use while in your downloads folder, make a habit of preferring typing the path to the script (/path/to/venv/bin/pip) rather than the module (/path/to/venv/bin/python -m pip).

In general, just avoid ever having ~/Downloads as your current working directory, and move any software you want to use to a more appropriate location before launching it.

It’s important to understand where Python gets the code that it’s going to be executing. Giving someone the ability to execute even one line of arbitrary Python is equivalent to giving them full control over your computer!

Why I wrote this article

When writing a “tips and tricks” article like this about security, it’s very easy to imply that I, the author, am very clever for knowing this weird bunch of trivia, and the only way for you, the reader, to stay safe, is to memorize a huge pile of equally esoteric stuff and constantly be thinking about it. Indeed, a previous draft of this post inadvertently did just that. But that’s a really terrible idea and not one that I want to have any part in propagating.

So if I’m not trying to say that, then why post about it? I’ll explain.

Over many years of using Python, I’ve infrequently, but regularly, seen users confused about the locations that Python loads code from. One variety of this confusion is when people put their first program that uses Twisted into a file called twisted.py. That shadows the import of the library, breaking everything. Another manifestation of this confusion is a slow trickle of confused security reports where a researcher drops a module into a location where Python is documented to load code from — like the current directory in the scenarios described above — and then load it, thinking that this reflects an exploit because it’s executing arbitrary code.

Any confusion like this — even if the system in question is “behaving as intended”, and can’t readily be changed — is a vulnerability that an attacker can exploit.

System administrators and developers are high-value targets in the world of cybercrime. If you hack a user, you get that user’s data; but if you hack an admin or a dev, and you do it right, you could get access to thousands of users whose systems are under the administrator’s control or even millions of users who use the developers’ software.

Therefore, while “just be more careful all the time” is not a sustainable recipe for safety, to some extent, those of us acting on our users’ behalf do have a greater obligation to be more careful. At least, we should be informed about the behavior of our tools. Developer tools, like Python, are inevitably power tools which may require more care and precision than the average application.

Nothing I’ve described above is a “bug” or an “exploit”, exactly; I don’t think that the developers of Python or Jupyter have done anything wrong; the system works the way it’s designed and the way it’s designed makes sense. I personally do not have any great ideas for how things could be changed without removing a ton of power from Python.

One of my favorite safety inventions is the SawStop. Nothing was wrong with the way table saws worked before its invention; they were extremely dangerous tools that performed an important industrial function. A lot of very useful and important things were made with table saws. Yet, it was also true that table saws were responsible for a disproportionate share of wood-shop accidents, and, in particular, lost fingers. Despite plenty of care taken by experienced and safety-conscious carpenters, the SawStop still saves many fingers every year.

So by highlighting this potential danger I also hope to provoke some thinking among some enterprising security engineers out there. What might be the SawStop of arbitrary code execution for interactive interpreters? What invention might be able to prevent some of the scenarios I describe below without significantly diminishing the power of tools like Python?

Stay safe out there, friends.


Acknowledgments

Thanks very much to Paul Ganssle, Nathaniel J. Smith, Itamar Turner-Trauring and Nelson Elhage for substantial feedback on earlier drafts of this post.

Any errors remain my own.


  1. Restricting which sites can drive-by drop files into your downloads folder is a great security feature, except the main consequence of adding it is that everybody seems to be annoyed by it, not understand it, and want to turn it off