"pffft. twisted isn't specific."The original goal of the Twisted project, as I have been frequently reminded of late, is to create a general, inter-operable mainloop that isn't specific to any particular protocol. The main loop wasn't a goal in itself, as the point of making it general was to provide an opportunity for all protocols could have serious, production-quality implementations that any Twisted application could have access to. Twisted itself ships with many different protocol implemenations in furtherance of this goal, in an attempt to get critical mass.
— W. Allen Short
This generality is a great strength. It means that we've attracted a small crowd of generalists. We have an excellent development process, ever-increasing quality of both code and documentation, and a wide variety of different protocol implementations and libraries for doing common networking and inter-process communication tasks. We have recently been lucky to attract a few more excellent developers to help with this.
The one thing we haven't been so lucky about is attracting specifists. Although we still need more people to make Twisted awesome as a library, our community is getting better and better at doing that. What we need even more than that is individuals with a very specific, focused interest on just one thing that Twisted does. Czars, if you will, to push the development of Twisted as a suite of interoperating applications.
Twisted already has within it the seeds of excellent replacements for Apache httpd, OpenSSH, BIND, hybrid ircd, Sendmail, imapd, pop3d, and a few other servers, not to mention clients like Pidgin and the OpenSSH command-line client. In order to sprout and take root, those seeds each need a dedicated advocate, someone who cares deeply about the experience of a user or administrator who just wants Twisted to perform one particular function and doesn't want to write their own application code to make it do that.
Projects like the ones above - OpenSSH and BIND, for example - have an advantage in becoming useful: they have dedicated people who care deeply about satisfying a particular use-case, and are singularly focused on that case. Since they only have the one problem to worry about, they can give it a much more direct treatment.
However, given the team of infrastructure programmers already working on Twisted, such a focused individual would have an incredible force multiplier. Consider the statistics on Conch from our 2003 USENIX paper on Twisted: going just by line count, Conch was 4x easier to write than even J2SSH, which was itself substantially smaller than OpenSSH. It was 10x easier to write than OpenSSH. So, with the support of Twisted as infrastructure, one Twisted application programmer can do the work of ten merely mortal ones ;-).
It might seem to those of you looking to write a chat client, DNS server, or whatever open-source giant that you want to do battle with, that Twisted is just a library, and you want to write an application. But we really want twisted to be a comprehensive suite of applications, we're just stretched too thin already to make it realize that potential.
So please rest assured that we would love to have your help with turning Twisted itself into a worthy competitor for these open-source giants - or, for that matter, if you want to build your own competitor as a layer on top of Twisted (for whatever reason: you love .ini files and we don't, you want a more freewheeling development process, or you want a different shade of green on your web pages) we'd still love to help you out and support that effort by fixing whatever issues you have with Twisted's core or protocols. There's even a super-project on Launchpad for Twisted-but-not-part-of-Twisted projects. I invite all you application developers out there to join that group and help us with world domination.
(If all that stuff about being ten times more effective as a programmer wasn't enough for you, how about this? On the Twisted Matrix Labs map of the post-revolutionary world, I'm pretty sure the Emancipated Territory of New Jersey is still missing an archduke and several viscounts. I can't make any promises, but if you get in on the ground floor of this thing there's still a chance you could be a ruling member of the Twisted over-government!)
Posts
9 comments:
Screw that. I want to be head of the secret police. I'll root out threaded application developers wherever they may lie.
from twisted.web import reactor
...would be a nice start.
Erm... you just want the name to be different, or what? the HTTP library does not seem, at first blush, to be a reasonable place to place the global reference to the event loop.
It's not entirely obvious to me how minor API changes achieve the goal of recruiting specialists to polish different areas of the project. I'm all for improving the clarity of APIs, but is this point really related to this post?
Glyph said "...In order to sprout and take root, those seeds each need a dedicated advocate, someone who cares deeply about the experience of a user or administrator who just wants Twisted to perform one particular function and doesn't want to write their own application code to make it do that."
Let's take HTTP as an example of "one particular function". Learning how Twisted can do DNS or chat or SMTP doesn't help me write an HTTP application; instead, it gets in the way. Any abstractions which serve the Twisted implementation over my user experience as an author of an HTTP application get in the way. Someone who cared deeply about my experience as a user would work to let me import everything I needed from a web-specific package, so I wouldn't have to remember that the Site class is in twisted.web but the reactor is in twisted.internet--that's a layer of abstraction which I as an HTTP application author shouldn't have to deal with.
Now, you *could* make a new specialized layer on top of twisted which did that, but IMO you don't need to; just allow the reactor to be importable from the twisted.web namespace. Ditto for the Site class, actually, since it's almost universally necessary.
That's nowhere near sufficient to make the Twisted HTTP experience more about my application than your framework. But it's a good starting point, for some specialist who truly cared about the user experience.
> That's nowhere near sufficient to make the Twisted HTTP experience more about my application than your framework. But it's a good starting point, for some specialist who truly cared about the user experience.
This is probably another documentation issue. For writing a straight up HTTP app, it's very likely you'll never have to import the reactor. Lots of the web examples do it anyway, though. Improving the documentation so that there is an excellent new-user experience is definitely one of the kinds of polish that would make Twisted Web a much more successful competitor to the other projects out there. Now, to find someone to do that work... :)
The problem with Twisted is not technical, its social. People don't know where to start.
If I may digress for a minute, I was talking recently with a friend about an interesting series of books that we both read. I was explaining how much I loved that the plot got larger as the books progressed.
My friend replied, "Yes, but what I liked is that the plot started out small. It started out with a few people you cared about, and then grew once you already knew the characters."
The problem with Twisted is that the plot doesn't start out small, with characters that you care about. Yes, I know about the finger server tutorial. The problem is that 1) finger servers aren't what people care about, and 2) its not clear how to progress from the finger server to the problem that I am trying to solve, today.
This post, for example, tells me that Twisted has lots of possible applications... but it doesn't tell me where to start. There is (in some ways) too much emphasis on the higher levels in the stack. If twisted is, at bottom, an event loop, then give me a way that I can start with just the event loop.
What is the most basic thing you can do with an event loop? Have some "container" of code, and have it get called when something happens. But I really don't know how to import a reactor, implement a container, and go.
If I were writing a tutorial on Twisted, I would do it as follows:
1. The big idea: Tell me when I need to do something.
In most applications, you spend a lot of time waiting. Or your code does, at least. You have a piece of code that does something important when a button is pushed, when a network request comes, or when there is new data to process.
Twisted is plumbing. It allows you to write that one thing that is important to your application, and then it makes sure that your code is called at the right time.
For example:
class http_server()....
[Blocking, simple server. Show that there is a bunch of looping/waiting going on - and the code for that dominates returning the "Hello World" page.]
Wouldn't it be nice to have this?
class http_server()...
[Simple as possible. have it receive a request, return the hello world page.]
The difference between #1 and #2 is that in #2, we decided to let twisted take care of all the plumbing.
2. Blocks of code.
[The point of this section is to show the simplest possible code container that does something. It is a glorified function: it gets inputs, it does stuff, it returns outputs. You can steal - and simplify - the inbox/outbox concept of Kamaelia. Something comes in, you handle it, you send it on. Use something like corotwine to do it blocking, first. Then show that you can trivially adjust it to do things in nonblocking way... and that is the point! No more waiting. Your code is called when it is needed.]
3. Lets build it up, add some options...
Now, start adding options, factoring, etc.
If I had this tutorial, I would know a lot easier where to start with Twisted.
@VanL This is exactly what is needed, as a potential twisted user, the shear amount of documentation is just boggling, when I last looked into it I was put off as I couldn't see where to start.
The opposite of this has to be the django project, within a short time in the tutorials you have something useful to work with.
>> I was put off as I couldn't see where to start.
Yup. I inherited an app and am constantly wondering why the heck anything was done anywhich way and, no, a finger tutorial doesn't do me any good. I haven't even thought about finger in more years than I care to count.
>> The opposite of this has to be the django project, within a short time in the tutorials you have something useful to work with.
Yes, except when you try to go beyond that. I've heard lots of "Ok, I did the tutorial and got the awesome polling, or blogger or whatever, now what?"
Twisted's problem is much bigger, though, because it's so wide that the documentation for hooking together the disparate pieces is sorely lacking.
I've also found a lot of the documentation and examples I've tried were of the "doesn't work with current versions" variety.
I'd love to help but all I could do is serve as a guinea pig for tutorials about the stuff I actually care about (mostly web client, ajax and, the comet client and server that I wish I could figure out how to use from Twisted but can't get off square one with).
For me it's not that I don't understand the programming model as it seems it is for many new users, it's the "how do I hook this to that and why do it that way" that I stumble over, if there's anything there doc-wise to even stumble on at all.
Post a Comment