Deciphering
Glyph
( )
Making Twisted Specific

Thu 24 September 2009

"pffft. twisted isn't specific."
          — W. Allen Short
The original goal of the Twisted project, as I have been frequently reminded of late, is to create a general, inter-operable mainloop that isn't specific to any particular protocol.  The main loop wasn't a goal in itself, as the point of making it general was to provide an opportunity for all protocols could have serious, production-quality implementations that any Twisted application could have access to.  Twisted itself ships with many different protocol implemenations in furtherance of this goal, in an attempt to get critical mass.

This generality is a great strength.  It means that we've attracted a small crowd of generalists.  We have an excellent development process, ever-increasing quality of both code and documentation, and a wide variety of different protocol implementations and libraries for doing common networking and inter-process communication tasks.  We have recently been lucky to attract a few more excellent developers to help with this.

The one thing we haven't been so lucky about is attracting specifists.  Although we still need more people to make Twisted awesome as a library, our community is getting better and better at doing that.  What we need even more than that is individuals with a very specific, focused interest on just one thing that Twisted does.  Czars, if you will, to push the development of Twisted as a suite of interoperating applications.

Twisted already has within it the seeds of excellent replacements for Apache httpd, OpenSSH, BIND, hybrid ircd, Sendmail, imapd, pop3d, and a few other servers, not to mention clients like Pidgin and the OpenSSH command-line client.  In order to sprout and take root, those seeds each need a dedicated advocate, someone who cares deeply about the experience of a user or administrator who just wants Twisted to perform one particular function and doesn't want to write their own application code to make it do that.

Projects like the ones above - OpenSSH and BIND, for example - have an advantage in becoming useful: they have dedicated people who care deeply about satisfying a particular use-case, and are singularly focused on that case.  Since they only have the one problem to worry about, they can give it a much more direct treatment.

However, given the team of infrastructure programmers already working on Twisted, such a focused individual would have an incredible force multiplier.  Consider the statistics on Conch from our 2003 USENIX paper on Twisted: going just by line count, Conch was 4x easier to write than even J2SSH, which was itself substantially smaller than OpenSSH.  It was 10x easier to write than OpenSSH.  So, with the support of Twisted as infrastructure, one Twisted application programmer can do the work of ten merely mortal ones ;-).

It might seem to those of you looking to write a chat client, DNS server, or whatever open-source giant that you want to do battle with, that Twisted is just a library, and you want to write an application.  But we really want twisted to be a comprehensive suite of applications, we're just stretched too thin already to make it realize that potential.

So please rest assured that we would love to have your help with turning Twisted itself into a worthy competitor for these open-source giants - or, for that matter, if you want to build your own competitor as a layer on top of Twisted (for whatever reason: you love .ini files and we don't, you want a more freewheeling development process, or you want a different shade of green on your web pages) we'd still love to help you out and support that effort by fixing whatever issues you have with Twisted's core or protocols.  There's even a super-project on Launchpad for Twisted-but-not-part-of-Twisted projects.  I invite all you application developers out there to join that group and help us with world domination.

(If all that stuff about being ten times more effective as a programmer wasn't enough for you, how about this?  On the Twisted Matrix Labs map of the post-revolutionary world, I'm pretty sure the Emancipated Territory of New Jersey is still missing an archduke and several viscounts.  I can't make any promises, but if you get in on the ground floor of this thing there's still a chance you could be a ruling member of the Twisted over-government!)