New Release

Yesterday I published a new release of DBXS for you all. It’s still ZeroVer, but it has graduated from double-ZeroVer as this is the first nonzero minor version.

More to the point though, the meaning of that version increment this version introduces some critical features that I think most people would need to give it a spin on a hobby project.

What’s New

• It has support for MySQL and PostgreSQL using native asyncio drivers, which means you don’t need to take a Twisted dependency in production.

• While Twisted is still used for some of the testing internals, Deferred is no longer exposed anywhere in the public API, either; your tests can happily pretend that they’re doing asyncio, as long as they can run against SQLite.

• There is a new repository convenience function that automatically wires together multiple accessors and transaction discipline. Have a look at the docstring for a sense of how to use it.

• Several papercuts, like confusing error messages when messing up query result handling, and lack of proper handling of default arguments in access protocols, are now addressed.

It’s A Good Time To Contribute!

If you’ve been looking for an open source project to try your hand at contributing to, DBXS might be a great opportunity, for a few reasons:

1. The team is quite small (just me, right now!), so it’s easy to get involved.

2. It’s quite generally useful, so there’s a potential for an audience, but right now it doesn’t really have any production users; there’s still time to change things without a lot of ceremony.

3. Unlike many other small starter projects, it’s got a test suite with 100% coverage, so you can contribute with confidence that you’re not breaking anything.

4. There’s not that much code (a bit over 2 thousand SLOC), so it’s not hard to get your head around.

5. There are a few obvious next steps for improvement, which I’ve filed as issues if you want to pick one up.

Share and enjoy, and please let me know if you do something fun with it.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics such as “How do I shot SQL?”.

It occurs to me that the lack of a standard, well-supported, memory-efficient interface for BLOBs in multiple programming languages is one of the primary driving factors of poor scalability characteristics of open source SaaS applications.

Applications like Gitlab, Redmine, Trac, Wordpress, and so on, all need to store potentially large files (“attachments”). Frequently, they elect to store these attachments (at least by default) in a dedicated filesystem directory. This leads to a number of tricky concurrency issues, as the filesystem has different (and divorced) concurrency semantics from the backend database, and resides only on the individual API nodes, rather than in the shared namespace of the attached database.

Some databases do support writing to BLOBs like files. Postgres, SQLite, and Oracle do, although it seems MySQL lags behind in this area (although I’d love to be corrected on this front). But many higher-level API bindings for these databases don’t expose support for BLOBs in an efficient way.

Directly using the filesystem, as opposed to a backing service, breaks the “expected” scaling behavior of the front-end portion of a web application. Using an object store, like Cloud Files or S3, is a good option to achieve high scalability for public-facing applications, but that creates additional deployment complexity.

So, as both a plea to others and a note to myself: if you’re writing a database-backed application that needs to store some data, please consider making “store it in the database as BLOBs” an option. And if your particular database client library doesn’t support it, consider filing a bug.