A Chicken in Every Pot and a Python on Every Port

Tuesday June 23, 2009
Twisted Matrix Labs is bent on world domination.  We spend so much time working at the level of fine-grained minutæ that we sometimes forget the overarching plan.  So here's a step back: what is Twisted for?

Most people know at least part of Twisted's origin story.  I was working on a text-based game, and I wanted a networking layer, and discovered that there was really nothing available.  I decided to write something general to base the game's networking core on, so I would be able to use production-quality protocols rather than toy "just for this game" stuff.

However, it wasn't just about the game.  That's a good thing, too, because the game has been falling behind quite a bit.  My game was just one example of code that you might want to write that could talk to a network, and my frustration was that despite large amounts of code being written to talk to networks, very little of it was directly usable by other code, and even less of it could be combined.  A major culprit here is that most networking software is written in C, where there is a stark contrast between "application" and "library"; a conscious, deliberate effort has to be made to expose functionality as a library, both in the code and in the build process.

Then there's the security situation.  A 2007 analysis of different types of vulnerability reports that buffer overflows were only recently overtaken by web application attacks, but are still the #2 for vulnerabilities overall, and #1 for OS vendor advisories.  Again, why is everybody still using all this network software written in C?  You can't even have a buffer overflow in most high-level languages.  (The even more depressing thing here is that, as the web development community has moved to higher-level languages, the majority have moved to the worst possible high-level language.  The vulnerability listings for web applications in that same report mostly have to do with flaws in PHP.)

So, the goal of Twisted is to provide a high-quality, high-level, secure implementation of every protocol spoken on the Internet.  We've achieved a lot, but there's still a long way to go.  Netcraft no longer seems to have any data on Twisted, because it is too low in the "other" category.  There's no site I'm aware of that does server market-share for DNS servers, but I'm betting that Twisted remains low in this category as well.

I believe Twisted remains popular in a growing segment of the network applications market, that is to say, applications that don't fit neatly into a single protocol.  If you want to control DNS, HTTP, SIP, and XMPP from a single program, it's far easier in Twisted than in anything else.  However, I think we can do better.   I want Twisted to take on BIND, Apache, Asterisk and jabberd directly as a server in its own right, not an integration mechanism or library.

One major area where Twisted is lacking is in focused, purpose-specific developers.  Apache has lots of people who are only interested in HTTP, Asterisk has people who are only interested in SIP, and libpurple has lots of people who are only interested in chat.  Twisted, by contrast, has excellent generalists, but few individuals to focus on the individual details of a single application.  I'm not sure how to recruit people who have that kind of monomaniacal focus to maintain individual components.  I think it's the details that such people would notice which is holding us back from being more competitive in the general server "market", such as it is.

This is a chicken-and-egg problem.  People interested in chat clients will often find libpurple before they find Twisted Words; people interested in web servers will often find Apache before they find Twisted Web.  Part of this is the lack of relevant conveniences and features, but probably an even bigger part is just our lack of a coherent web presence for those interest groups.  While I think that a lot of people looking for these things would be delighted to find something as easy to script and re-shape as Twisted is, they don't start out by looking for an omni-server platform.

So go update the web site, and take over the world!