Twisted Matrix Labs is bent on world domination. We spend so much time working at the level of fine-grained minutæ that we sometimes forget the overarching plan. So here's a step back: what is Twisted for?
Most people know at least part of Twisted's origin story. I was working on a text-based game, and I wanted a networking layer, and discovered that there was really nothing available. I decided to write something general to base the game's networking core on, so I would be able to use production-quality protocols rather than toy "just for this game" stuff.
However, it wasn't just about the game. That's a good thing, too, because the game has been falling behind quite a bit. My game was just one example of code that you might want to write that could talk to a network, and my frustration was that despite large amounts of code being written to talk to networks, very little of it was directly usable by other code, and even less of it could be combined. A major culprit here is that most networking software is written in C, where there is a stark contrast between "application" and "library"; a conscious, deliberate effort has to be made to expose functionality as a library, both in the code and in the build process.
Then there's the security situation. A 2007 analysis of different types of vulnerability reports that buffer overflows were only recently overtaken by web application attacks, but are still the #2 for vulnerabilities overall, and #1 for OS vendor advisories. Again, why is everybody still using all this network software written in C? You can't even have a buffer overflow in most high-level languages. (The even more depressing thing here is that, as the web development community has moved to higher-level languages, the majority have moved to the worst possible high-level language. The vulnerability listings for web applications in that same report mostly have to do with flaws in PHP.)
So, the goal of Twisted is to provide a high-quality, high-level, secure implementation of every protocol spoken on the Internet. We've achieved a lot, but there's still a long way to go. Netcraft no longer seems to have any data on Twisted, because it is too low in the "other" category. There's no site I'm aware of that does server market-share for DNS servers, but I'm betting that Twisted remains low in this category as well.
I believe Twisted remains popular in a growing segment of the network applications market, that is to say, applications that don't fit neatly into a single protocol. If you want to control DNS, HTTP, SIP, and XMPP from a single program, it's far easier in Twisted than in anything else. However, I think we can do better. I want Twisted to take on BIND, Apache, Asterisk and jabberd directly as a server in its own right, not an integration mechanism or library.
One major area where Twisted is lacking is in focused, purpose-specific developers. Apache has lots of people who are only interested in HTTP, Asterisk has people who are only interested in SIP, and libpurple has lots of people who are only interested in chat. Twisted, by contrast, has excellent generalists, but few individuals to focus on the individual details of a single application. I'm not sure how to recruit people who have that kind of monomaniacal focus to maintain individual components. I think it's the details that such people would notice which is holding us back from being more competitive in the general server "market", such as it is.
This is a chicken-and-egg problem. People interested in chat clients will often find libpurple before they find Twisted Words; people interested in web servers will often find Apache before they find Twisted Web. Part of this is the lack of relevant conveniences and features, but probably an even bigger part is just our lack of a coherent web presence for those interest groups. While I think that a lot of people looking for these things would be delighted to find something as easy to script and re-shape as Twisted is, they don't start out by looking for an omni-server platform.
So go update the web site, and take over the world!
Most people know at least part of Twisted's origin story. I was working on a text-based game, and I wanted a networking layer, and discovered that there was really nothing available. I decided to write something general to base the game's networking core on, so I would be able to use production-quality protocols rather than toy "just for this game" stuff.
However, it wasn't just about the game. That's a good thing, too, because the game has been falling behind quite a bit. My game was just one example of code that you might want to write that could talk to a network, and my frustration was that despite large amounts of code being written to talk to networks, very little of it was directly usable by other code, and even less of it could be combined. A major culprit here is that most networking software is written in C, where there is a stark contrast between "application" and "library"; a conscious, deliberate effort has to be made to expose functionality as a library, both in the code and in the build process.
Then there's the security situation. A 2007 analysis of different types of vulnerability reports that buffer overflows were only recently overtaken by web application attacks, but are still the #2 for vulnerabilities overall, and #1 for OS vendor advisories. Again, why is everybody still using all this network software written in C? You can't even have a buffer overflow in most high-level languages. (The even more depressing thing here is that, as the web development community has moved to higher-level languages, the majority have moved to the worst possible high-level language. The vulnerability listings for web applications in that same report mostly have to do with flaws in PHP.)
So, the goal of Twisted is to provide a high-quality, high-level, secure implementation of every protocol spoken on the Internet. We've achieved a lot, but there's still a long way to go. Netcraft no longer seems to have any data on Twisted, because it is too low in the "other" category. There's no site I'm aware of that does server market-share for DNS servers, but I'm betting that Twisted remains low in this category as well.
I believe Twisted remains popular in a growing segment of the network applications market, that is to say, applications that don't fit neatly into a single protocol. If you want to control DNS, HTTP, SIP, and XMPP from a single program, it's far easier in Twisted than in anything else. However, I think we can do better. I want Twisted to take on BIND, Apache, Asterisk and jabberd directly as a server in its own right, not an integration mechanism or library.
One major area where Twisted is lacking is in focused, purpose-specific developers. Apache has lots of people who are only interested in HTTP, Asterisk has people who are only interested in SIP, and libpurple has lots of people who are only interested in chat. Twisted, by contrast, has excellent generalists, but few individuals to focus on the individual details of a single application. I'm not sure how to recruit people who have that kind of monomaniacal focus to maintain individual components. I think it's the details that such people would notice which is holding us back from being more competitive in the general server "market", such as it is.
This is a chicken-and-egg problem. People interested in chat clients will often find libpurple before they find Twisted Words; people interested in web servers will often find Apache before they find Twisted Web. Part of this is the lack of relevant conveniences and features, but probably an even bigger part is just our lack of a coherent web presence for those interest groups. While I think that a lot of people looking for these things would be delighted to find something as easy to script and re-shape as Twisted is, they don't start out by looking for an omni-server platform.
So go update the web site, and take over the world!
Posts
20 comments:
I'd love to update my site to Twisted. Where can I find Twisted hosting company that I can afford?
@Michael: webfaction.com?
Hi Glyph,
I've been recently using twisted and I think you need to do one single thing in order to achieve the world domination you so crave:
Clear up the twisted core (defer, reactor, task) and so on into a small independent library that people can just drop in in their applications when they want to do async stuff.
Right now there are a lot of interesting bits of functionality in twisted that are hidden behind weird names, no documentation, and the fact that twisted is 15MB, uses Java style names and requires almost certainly a rewrite of your application.
If people when need to do async stuff think first "I'll use twisted" instead of "I'll use threads", then you're a step closer!
I meant to write this in a more expanded version as a blog post, but given you're calling for help here, thought it'd be more relevant.
I have been looking forward to imaginary for years.. I am still looking for a good mud style application server in python. For now still stuck with LambdaMoo.
twisted needs a way to scale better on multi-core server class hardware. and that isn't going to happen until Python gets rid of the GIL.
I have pretty much started doing all my new server dev work in Erlang/OTP.
It is a framework very much in the "SPIRIT" of what Twisted tries to accomplish.
I'm not a genius when it comes to this stuff, but can't you take advantage of multiple cores by running multiple server processes? Not that would entirely solve the scaling issue...
@Michał: tummy.com, slicehost.com, linode.com?
@Ορέστης: We already did. That library weighs in at 1.2M and is known as "Twisted Core". The fact that you didn't know this rather proves my point about the web site :). (There are problems with setuptools and the cheeseshop which make it difficult for us to provide an easy_installable version of the core, but you can easy_install Twisted itself, so the only problem is bandwidth.) As far as "uses Java style names" - not true. We use Smalltalk style names. The fact that Java does too is just a coincidence :).
@ToddB: Sorry to disappoint. Please join us in #imagination on freenode to talk about it :).
@Jarrod: If you want multiprocessing with Twisted, use Ampoule. In the future I hope that we'll include it in the core.
@Wyatt: Right on. See my response to Jarrod :).
@glyph Thanks, didn't know about it. However, I'm talking about a more in-depth rewriting. Things like
from twisted.core import reactor, defer
It makes little sense to me why should reactor be placed in 'twisted.internet' - it can be used in places where no internet is visible.
I appreciate this is going to be a huge effort, but other projects have managed to pull it off (eg. django with its magic-removal branch) and it helped a lot. It also has to break backwards compatibility, which is always an issue.
Of course, twisted has also the burden of actually teaching people about async programming (the same way that django has taught many people about web programming), so the documentation *really* needs to be top-notch.
Does Twisted Matrix Labs' desire for world domination extend to the realm of web frameworks? In other words, do you want a Twisted-based web framework such as Mantissa to compete with Django, Rails, and the like? Or are you content to let Twisted Web be a container for WSGI-enabled web frameworks such as Django and Pylons?
I've wanted to use twisted for years. But I can never wrap my head around it. Even bought the book a few years ago and even read it. Still my brain just doesn't work that way typically.
However I will say after messing with the multiprocessing module and tracking down deadlocks for the last week or so(my fault not the modules), i may now be ready to dig in to twisted a little more since I now am trying to solve problems that twisted might help solve.
So perhaps twisted was ahead of it's time, maybe it's time has arrived?
Also, to keep with the fashion of bitching about the GIL and being all trendy I'd like to say "I hatez the GIL it is teh sux...I'll use erlang LOLOLOL!!!!one!111"
Haterz go /home! Erlang is teh true sux0rs. It's not as fun() as They'd have you Believe.
[j/k]
Twisted is amazing, but scary. It's a complex beast with weird naming and very little documentation.
An API re-haul is probably not needed, but better docs desperately is.
@Ορέστης: It sounds like you're not talking about "reworking", but rather a renaming of a few modules. I don't necessarily disagree: I hope one day to make a new package, 'twisted.loop', which has most of the main-loop functionality from twisted.internet, because I think that would be a clearer name. Also, making a new package would allow us to be very deliberate in exposing fewer names, making the API documentation more readable.
However, I think better documentation of the existing API is more worthwhile in terms of effort than small tweaks to the naming of modules or APIs.
@Matt Campbell: Both! :-). We want Twisted to be the container of choice for other web frameworks, but of course it would be great for Nevow to get more adoption as well. For now, I'm more concerned with making more "traditional" web frameworks (django, turbogears, zope, etc) feel welcome and well-supported on Twisted than promoting our particular web-framework religion. While I want twisted.web and Nevow to be awesome, I want authors of WSGI frameworks to know that they will never be second-class citizens in the Twisted universe.
I'd have to agree: first step is documentation, perhaps with particular focus on how to get from existing code to twisted (it's easy enough to start from scratch in twisted, but not as easy as throwing together some battery-based python code without it and *then* realizing you need more...)
Part of that should be making clear how incomplete (or not) some of the modules are (last I looked twisted.words didn't support any actual security - so either you have a feature gap or a marketing/documentation gap :-)
I also have gone through several twisted books, but there's still a lot of mental friction to switch gears into the twisted mindset (even for someone with unix kernel internals experience...)
Maybe you already know about all of the following, but here's what I'd need to use Twisted Web in production instead of Apache on my employer's main Web servers:
Excellent FastCGI container, so it can host PHP well. Yes, we (currently) run a mix of PHP and Python.
Ability to make configuration changes that are persistent but applied immediately.
Something like mod_wsgi in daemon mode, which automatically spawns and manages WSGI application processes, as different users if desired.
Mature reverse proxy implementation (maybe this already exists).
And even then, we probably wouldn't use Twisted Web on our main web servers unless it were widely known among system administrators, including those at our managed hosting provider (Peer 1).
I don't mean to rant; I just want to provide an additional perspective on what it will take to achieve world domination.
Oops, "FastCGI container" probably isn't the correct term. I mean something that can spawn, manage, and communicate with FastCGI processes, like mod_fcgid for Apache.
Your path is clear: make twisted.web the obvious front-runner for WSGI, ahead of paste, cherrypy, etc.
I agree with @autoloquist. What percentage of developers come to Twisted for Web stuff? I've been watching Twisted Web/Web2 for the last couple years, and the "Web Development with Twisted" wiki page always scares me away. I'm not sure, but it's possible that if the Web story was less scary, it could be a gateway into the larger Twisted Matrix.
I'd very much like to see twisted world domination, yay!
Things i can imagine, will be in the way of this goal:
- better documentation (tutorials, examples)
- confusing flux with web, web2, nevow
- We need more books then one!
Also, i think mantissa could lead to world domination right away, but again, it's so poorly documented that i can't even get examples to work because they are too old :(
I love to use twisted for nearly everything i do in networked programming, apart from the html/ajax side of things (imo, old nevow livepage was much much simpler then the current fragment, javascript approach. It's too complicated, hard to learn, where other frameworks are easier. won't use that. Again, mantissa could be so great!).
@_Mark_, @Wyatt, @Buster, @Lucian: Yes, yes. I said documentation is important in the post. Fix doc bugs! Contribute patches! :)
@autoloquist: Absolutely. We've been hard at work on that. The 9.0 release should be much, much better.
@Matt Campbell: I definitely understand the need for more and better containers for different ways that we can interface to other web technologies. However, I wonder if you could clarify on one point: what do you mean by "configuration"? Right now, twisted.web's flexibility is both its greatest strength and its weakness; its configuration can be a complete Python program, which has obvious problems with persistence.
@Buster: Thanks for your interest in Mantissa! :-D. I understand your criticism, and we've been working to improve the documentation.
Post a Comment