It's somewhat official now, so I guess I have to announce it: Divmod is doing a massive refactoring of our application, starting with the database. Work began a week ago in my Quotient sandbox, and has been continuing around the clock since then.
Progress is now visible in a more public location:
Everybody knows that's
stupid. Plus, we are mostly migrating our old code base and cleaning it
up along the way; the only component getting a complete rewrite is
Atop - the rewrite of which is so fundamentally better that we came up with
a better name: "Axiom".
To clarify the naming situation: the new Divmod repository has Axiom
and a new version of Mantissa, Python package name 'xmantissa' to avoid
package name conflicts during the transition period. eventually it will
contain a package for Quotient, Sigma, and several other things. During the
transition the Python package names will all start with 'x' but I will still
refer to them by their project names, since the older projects will go away
and the module names will eventually change.
Initially I was very concerned when we began the experiment that lead to
this code revolution. It began, as bad
ideas are wont to do, as a joke. I mentioned some client work that we
are doing to JP,
(who had already been rewriting some
things) and my difficulty in choosing an appropriate persistence
solution given some of the maintenance issues we'd been having with
Exasperated after an hour of discussion, JP said, "Why don't you just use
SQLite". Now, I'd looked into it some time ago and (the ostensible punchline
of the joke) it was garbage. However, that was SQLite version 1, not
SQLite3, which has a different API and several critical features that made
it considerably more appropriate to our tasks.
A few hours later I had a working prototype of maybe half of the
functionality from our existing database. I was suitably impressed; SQLite
was giving us all the benefits of SQL (ad-hoc queries and indexing,
relational operations on data, a fast query engine) without any of the
drawbacks (difficult to customize, unportable server, fragile and
time-consuming deployment). I realized that we had got something radical on
In the past I've been very conservative about telling people when and
whether to use Divmod's open-source released software, With this new system,
I say: jump in. Use it. Only a week into implementation it might be a
bit premature to launch a production system with it just yet, but
indications are very positive that we will be able to do just that within
The code is shorter, clearer to read, easier to maintain, and the database
is Pickle Free℠.
(Okay, it's not really a service mark, but it should be. Pickle is
the winner for causing problems for us.)
We are building from our experience with 5 previous persistence systems, 3
previous plug-in frameworks, and 4 previous authentication databases. A
curious side-effect of all that experience - and the effect that the title
is referring to - is that certain development methodology concepts become
irrelevant. Most notably, "YAGNI", from XP, is no longer of any use: we know
exactly how much extensibility we need. At every point in
implementing this system we have known whether to fuse a component together
because we'd built unnecessary additional complexity into previous systems,
and where to use a plug-in architecture because we'd needed to inject ugly
code into the middle of a monolithic routine.
As a result, where our architecture was heavily monolithic before, now it is
almost entirely composed of plugins. It is so plugin-happy, in fact, that
there is a database with Service plugins in it, which activate when the
database is started from twistd; it contains its own configuration,
including port-numbers, so nothing need live in a text configuration file.
The web application system is built around this as well; so there is a
plugin lookup for invoking raw nevow IResource implementors without sessions
(for example, for XML APIs), then IResource implementors which do require a
session, then IResource implementors which are specific to a particular
user, and finally Fragment instances which plug into a generic hierarchical
navigation system. At each level there is a distinct and clear place to put
new plugin code, and large portions of it are self-similar. For example, you
can install the hierarchical navigation both onto a public site and onto a
user's private application pages, since the "web server" implementation is
an IResource which can be installed either onto a toplevel database, or any
user's personal database.
Oh, and did I mention - LivePage support built right in?
The net result of this is that you can build themeable, multi-user web
applications with the code that's in SVN right now. The example
isn't visually appealing, but the code is nice, and it's composed from a
stack of plugins.
I'm very excited about the possibilities of what we'll be transforming our
system and our application into within the next few weeks. I'd like to
invite everyone who has been interested in Divmod's open source work in the
past to have a look at the new repository, and consider coming to #divmod on
irc.freenode.net to look for something to hack on. Considering the
higher-level and easier-to-understand nature of the implementation of Axiom
vs. Atop, I would also love it if we could find some people to help us
document it right from the start.
So - anybody out there looking for an open source project's website to