Threat 1: Attacks From The Outside

This article continues my series on my personal threat model for the internet.  In this article, I'm going to talk about the threat of automated attacks coming in to your computer over the internet, while it is connected to the internet.

The basic problem underlying this threat is the same as that underlying threats #2 (malicious e-mail messages which attack your e-mail program) and #3 (malicious web pages which attack your web browser): the software you are running on your computer, which you need to do your job, play your games, or otherwise get value out of your computer, is full of bugs.  Some of those bugs are security problems.  The most dangerous type of security problem is one that allows some data which a program is reading, which is supposed to just be processed by the program, to overwrite portions of that program's memory such that it takes over the program.  That data is then itself a program, and can take over your computer.  Unfortunately, this type of problem is very common.

The first thing you need to do to protect against these threats is to regularly install security updates for your computer.  On Windows you can do this by using Automatic Updates, on MacOS X it will be done for you by Software Update, and on Ubuntu, Update Manager.

When updates are available, make sure to install them as soon as you can!  By the time an update is available, the problem that the update is intended to fix has often been made public already.  The publication of the problem allows the update to be created in the first place, but it also allows malicious individuals to create attacks from it.  The longer you wait, the longer you are vulnerable to problems which have been made public, and thus can be exploited by the largest population of attackers.

However, even if all of your software is fully up-to-date, it still isn't perfect.  The general strategy for dealing with this type of problem, then, is to make sure that only data from sources you trust will ever be allowed into that software.  This limits your exposure to attacks.

In later posts I'll talk about limiting your exposure to malicious data that you have specifically requested, but right now I'm just going to talk about preventing unsolicited data getting to your computer directly over the internet.  The best way to do this is to get a commodity hardware router, and put it between your computer and the internet.  Devices such as this are made by vendors such as linksys, belkin, buffalo or netgear.

You don't need to get a router with fancy "security" features like an "SPI firewall" or "intrusion detection".  In my opinion these features don't add a lot - in fact, they will often cause difficult-to-diagnose problems for home users.  Of course, the people who sell these devices love to put the word "security" on the box as many times as possible, but you really only need the most basic security feature, and that's the one that isn't really a "security" feature at all.

The basic feature that a router adds is a separate layer of protection, independent from anything you can do to your computer itself.  If your home computer is hooked up directly to the internet, it looks like this:

That is, whenever your computer tries to contact another computer on the internet, it sends a request directly via your modem.  Whenever another computer tries to connect to you, it goes directly to your computer.  This means that if there are programs that you don't know about, which your operating system vendor, or some application has left running on your computer, anyone on the internet will be able to access them.

If those programs were all perfectly secure, that would be fine.  Unfortunately, programmers make mistakes, and mistakes lead to bugs, and bugs sometimes lead to security problems.

When you have a router, the picture looks more like this:

which is to say, when your computer submits a request to another computer on the internet, the router sees that the request is coming from inside the network, and transparently forwards it to the outside, establishing a channel of communication.  However, when another computer tries to talk to the IP address that your ISP gives you, the device they find is the router.  The router itself is a very simple device, and, unless you've done something unusual to it, will never be running any programs beyond the ones necessary to move traffic between you and your network.  Because one of the functions of a router is to allow multiple computers on your home network, when connections come in from the internet, the router doesn't know which computer it should go to, even if you only have one.  So the incoming connection will be refused, never having a chance to get to your computer.

This is preferable to running "firewall" software on your computer, for two reasons:

1. Firewall software is still running on your computer, and thus on your operating system.  If your operating system itself has a flaw in it, the firewall can't protect you.
2. Software which listens for incoming connections is doing so for a reason.  Different components of the same program will sometimes communicate with each other over a network connection internal to the same computer - as a user of those programs, you really shouldn't need to know this.  Firewall software will present you with prompts to allow or deny permission for programs: these prompts often boil down to "do you want this to work?"  If you say yes, your computer will be exposed to a potential threat, if you say no, the program will break.

Of course, if you've prevented other people's computers from accessing yours, there are some programs which will now be unable to connect to your computer.  BitTorrent, for example, is notorious for performing poorly if other users can't connect to you directly.  Certain voice-over-IP programs will also have problems.  To address this, you can add rules to your router to allow specific incoming connections, without opening the floodgates to everything.  This is referred to as "port forwarding", and portforward.com is a good resource.  If installing a router causes any problems with network applications that you use, consult their documentation: port-forwarding issues are usually prominently covered early on.

My Threat Model

As a "computer guy", I am sometimes called upon by friends and family to opine on what makes a computer or a network secure.  Many of my colleagues are in the same situation.  As a "networking guy", I get similar questions from even from experienced "computer guys".

Users have very peculiar ideas about security.  Users — and I include myself in this grouping — will become confused even in areas of the computing experience where billions of dollars have been spent trying to make the experience as easy and comprehensible as possible.  So it stands to reason that users will often be confused in the area of security, by its nature the least usable and comprehensible area of computing.  Attacks are arcane, and, by definition, unexpected ways that software can be manipulated.  Yet, these attacks are very relevant to users, who want to understand what, exactly, they are vulnerable to and how to defend against it.

It's basically impossible to try to understand computer security this way, let alone explain it.

The important thing to remember in any security situation is this: what do you have of value, and what is the threat to it?  Computer security professionals call the answer to this question the "threat model".  Stephen Colbert calls it the ThreatDown.  No matter what you call it, it's important to enumerate the threats that you're defending against.  Any security measure that you take which is not designed to protect you from a threat which you can, at the very least, imagine and describe, is just extra cost.

In my case, people ask me about three broad classes of user:

1. users who have networked computers in a home, and use them for checking email, browsing the web, online shopping, and games,
2. users who have networked desktop computers in a business, and use them for email, web, and business applications, and
3. users who have networked server computers that are running server applications.

These users all have roughly similar threat models, so I'm going to lump them together for the sake of simplicity, with a nod to a few specific situations.

I believe there are five major types of attacks which threaten average users on the internet today.

1. Automated attacks that attempt to connect to your computer and exploit a flaw in its operating system or in software that is running a server, and install malicious software on your computer.
   
2. E-mail attacks, which attempt to deliver a message which will exploit a flaw in your desktop e-mail client to install malicious software on your computer.
   
3. Browser attacks, which attempt to get your browser (either with or without your consent) to visit a site which will exploit a flaw in your browser software to install malicious software on your computer.
   
4. Phishing attacks, which attempt to convince you to disclose information about yourself, such as bank account numbers, passwords, or personal details that can be used to access those other things.
   
5. Snooping attacks, which attempt to read information in transit between you and another computer.  Usually snooping attacks read passwords in an attempt to allow the attacker to impersonate you later.

Attacks 1-3 are all based on the same premise: software is flawed, and sometimes the flaws in it can be exploited to get it to do things that it should not do.  There are multiple resources under threat here: your computer itself (i.e. its processing power), your network connection, and the data stored on your computer.

Attacks 4 and 5 are in a different class.  They're attempting to get you to reveal information over the network, either with or without your knowledge.  The resource under threat here is the information you are transmitting - in most cases, the information being sought is a token which allows you access to some resource; anything from a username and password to your facebook account (which allows for stealing your personal information or impersonating you) to a debit card number (which allows attackers access to the money in your bank account).

I have fairly simple ways to protect yourself against each of these types of attack.  In a series of follow-up articles, I'll cover each of those strategies.  They should cover a wide variety of attacks with a minimum of effort and cost.  Of course, these defenses aren't perfect.  It's possible that someone who knows much more about security than I do will correct me, but if so, that's so much the better.

More importantly, I will try to provide simple abstractions that allow you to reason about each type of attack without understanding the intricacies of the technology involved.  A major reason I've decided to try to write about this is that security vendors play upon the intuitive (and wrong) understanding that most people have about computer security: equating it with physical security, making their security widget the digital "lock" for the digital "house" of your computer.

I am targeting this series at a fairly nontechnical audience.  I realize that my audience here mostly rates pretty high on the nerd spectrum; my hope is that you will agree with what I say sufficiently that this will be a useful resource for you to refer your less technical friends and family.  To maintain your interest, however, I'll also be embedding some details about the reasoning behind my own security practices.  See you next time!

Update: I accidentally posted a draft of this rather than a final copy; some of the sentences and paragraphs were incomplete.  I hope that I've now corrected this.

A Chicken in Every Pot and a Python on Every Port

Twisted Matrix Labs is bent on world domination.  We spend so much time working at the level of fine-grained minutæ that we sometimes forget the overarching plan.  So here's a step back: what is Twisted for?

Most people know at least part of Twisted's origin story.  I was working on a text-based game, and I wanted a networking layer, and discovered that there was really nothing available.  I decided to write something general to base the game's networking core on, so I would be able to use production-quality protocols rather than toy "just for this game" stuff.

However, it wasn't just about the game.  That's a good thing, too, because the game has been falling behind quite a bit.  My game was just one example of code that you might want to write that could talk to a network, and my frustration was that despite large amounts of code being written to talk to networks, very little of it was directly usable by other code, and even less of it could be combined.  A major culprit here is that most networking software is written in C, where there is a stark contrast between "application" and "library"; a conscious, deliberate effort has to be made to expose functionality as a library, both in the code and in the build process.

Then there's the security situation.  A 2007 analysis of different types of vulnerability reports that buffer overflows were only recently overtaken by web application attacks, but are still the #2 for vulnerabilities overall, and #1 for OS vendor advisories.  Again, why is everybody still using all this network software written in C?  You can't even have a buffer overflow in most high-level languages.  (The even more depressing thing here is that, as the web development community has moved to higher-level languages, the majority have moved to the worst possible high-level language.  The vulnerability listings for web applications in that same report mostly have to do with flaws in PHP.)

So, the goal of Twisted is to provide a high-quality, high-level, secure implementation of every protocol spoken on the Internet.  We've achieved a lot, but there's still a long way to go.  Netcraft no longer seems to have any data on Twisted, because it is too low in the "other" category.  There's no site I'm aware of that does server market-share for DNS servers, but I'm betting that Twisted remains low in this category as well.

I believe Twisted remains popular in a growing segment of the network applications market, that is to say, applications that don't fit neatly into a single protocol.  If you want to control DNS, HTTP, SIP, and XMPP from a single program, it's far easier in Twisted than in anything else.  However, I think we can do better.   I want Twisted to take on BIND, Apache, Asterisk and jabberd directly as a server in its own right, not an integration mechanism or library.

One major area where Twisted is lacking is in focused, purpose-specific developers.  Apache has lots of people who are only interested in HTTP, Asterisk has people who are only interested in SIP, and libpurple has lots of people who are only interested in chat.  Twisted, by contrast, has excellent generalists, but few individuals to focus on the individual details of a single application.  I'm not sure how to recruit people who have that kind of monomaniacal focus to maintain individual components.  I think it's the details that such people would notice which is holding us back from being more competitive in the general server "market", such as it is.

This is a chicken-and-egg problem.  People interested in chat clients will often find libpurple before they find Twisted Words; people interested in web servers will often find Apache before they find Twisted Web.  Part of this is the lack of relevant conveniences and features, but probably an even bigger part is just our lack of a coherent web presence for those interest groups.  While I think that a lot of people looking for these things would be delighted to find something as easy to script and re-shape as Twisted is, they don't start out by looking for an omni-server platform.

So go update the web site, and take over the world!

Why Phones Lost

This morning I was reading Antonio Rodriguez's "Path Dependence And Smartphones" post, where he muses about different perspectives on the "smartphone" market; in particular the European / American divide outlined by Tomi Ahonen in "A Tale Of Two Smartphones".  Antonio tends to think a few years ahead of his time, so I'm always interested in his take on trends like this.  It seems that he's very cautiously optimistic that the "user customized mobile computer" thing is an important trend, but he also notes that Mr. Ahonen believes that the "[smartphone] operating system and any applications had ZERO bearing on the decision [of which phone to buy]. Not for mass market consumers"; and maybe we're looking at this the wrong way.

As I started thinking about my own reactions to this, I realized: I've heard this tune before.  Remember when pundits used to talk about "convergence" between television and computers?  Since the advent of the computer, futurists have been predicting the dawn of a strange new device: part computer, part television, part telephone, part vacuum cleaner.  What would it look like?

Well, a few months ago, I feel like Paul Graham answered that question pretty definitively.  For years, we've wondered what you would get if you mixed computers and televisions.  In Mr. Graham's words: "We now know the answer: computers."

As a child of the digital age — I've been using computers with keyboards, mice, color displays, and networking almost as long as I've been able to read — I always found this conclusion somewhat obvious.  A few of the early computers that I had the opportunity to use, an Atari 800 and an Amiga 1000, both used televisions as monitors, so I have always thought of a television as an output device — you could plug it into a VCR, a computer, or a cable box, but fundamentally it was just a bag of pixels.

I remember the exact moment that it dawned on me that computers were going to take over from TV: I was 14 years old, playing Myst for the first time, and monkeying with the configuration of system extensions that were loaded on my computer in order to squeeze the last few ounces of performance so that the video clips in the game would play smoothly.  I remember thinking, "This is just a problem with RAM and CPU.  In a few years computers will have so much of both that you'll be able to play full screen video without even turning off any extensions."

I, uh, had a pretty limited idea of how optimization worked at the time (the video was still jerky even after I turned off all my extensions), but I am frequently reminded of this insight when I am watching YouTube movies on my LCD "television".  That television, by the way, is just a monitor for a computer that runs Ubuntu so I can watch Hulu and YouTube.  I think maybe I have cable bundled with my internet service, because it's cheaper that way but I've never plugged it in to anything.

I didn't realized how powerful articulating this particular idea is until recently though, because I didn't realize just how much money is spent protecting obsolete infrastructure from the relentless onslaught of microprocessor technology.  Phone companies — which, increasingly, are combination cable/phone/internet companies — are stuck between a rock and a hard place.  As Internet service providers, they are a facilitator of the transition, and make a huge amount of money selling network services to people to make their computers more useful.  But, as cable companies, they want people to think that television is some special, extra expensive thing that needs to be delievered over a different cable.  As phone companies, both wired and wireless, they want people to think that voice and SMS data are special, extra expensive things that need to be delivered via special, magical wireless signals that can't be reduced to the simple and banal "internet".  At the same time, especially as wired phone companies, they want the cost savings that comes from doing all of their networking as plain old IP, with no actual pesky phone circuits to worry about.  Except they still want to sell you the service as if the phone were a different thing from your "internet" connection.  (Whenever I see an ad for Comcast Digital Voice, I can't help but think, "Do you think that's air you're breathing?".)

There's still a lot of speculation in each of these industries that some new, hybridized technology is going to create a special and unique relationship with the consumer.  But that's one thing Mr. Ahonen got right: the consumer doesn't care about your "operating system".  They don't care about your "applications".  They just care what they can do with their technology, and they care how much it costs to do so.  The thing is, computers do more, and cost less, than any other specialized, dedicated technology.  If your industry is fighting computers in the hopes of holding on to some residual value, you are going to lose.  Here's a simple formula:

Computer + X = Computer

Consider a few specific examples: the convergence of computers with television has resulted in three general categories of technology: YouTube (and other flash video sites, such as Hulu), Tivo (and other DVRs), and digital cable boxes with on-demand technology.  YouTube is a program you run on a computer to watch videos.  A Tivo is a computer (running linux) that is running a program to let you watch and record videos.  And those cable boxes are computers (running some crappy cut-down embedded OS) that let you watch videos on the cable company's terms.  Whether or not your customers care about choice, all these things are computers because it's fundamentally cheaper and easier for the vendors to produce these things out of commodity PC components rather than specialized "media" electronics.

But Mr. Graham neatly outlined that trend already, so let's move on to other industries.  What happens when you add a computer to an accounting ledger?  You get a computer program (like BusinessMind, or QuickBooks) which lets you do accounting on your computer.  Computers and books?  The Kindle, which is a hand-held computer that lets you read books.  If you look a bit deeper, you'll find that the Kindle is actually a computer program that can run places other than its dedicated device.  Only crafty marketing folks prevent it from being more widely accessible; say, on your desktop or "television".

Let's get to the point of this whole schpiel: phones.  Phones are already computers, pure and simple.  They are just small computers with microphones and speakers, and soon, cameras and screens.  You can look at the exciting developments in the world of phones and see that this is so.  What are the hottest phones of the last few years?  The iPhone, which is a small Macintosh computer, and the G1, which is a small Linux PC.  Microsoft would have you believe that their small Windows PCs are equally relevant, even if they are clearly an also-ran in this category.  (Disclosure: I actually have a Windows Mobile phone, and I'm fairly happy with it, but I'll be glad when I can finally ditch it for Android.)  None of these "phones" does anything interesting in the area of phone-ness.  They don't have particularly awesome voice quality or particularly awesome reception or even particularly awesome voicemail, although the iPhone certainly raised the bar.  They're just better computers than the previous generation of "phones"; computers that can run a wider variety of programs.

However, phones are still computers with weird restrictions, restrictions that are purely a function of the "path dependence" that Antonio mentions, which dragged them out of the muck and the mire of the telecom industry.  SMS is my favorite example of this: 10¢ to send a 140 character message.  How much does a tweet cost on twitter?  How much does an instant message cost on AIM, or Google Talk, or any IRC network you please?  If you were billed at SMS rates to read this post, it would have cost you $10; the cost of a decent paperback.  I know I'm wordy, but I'm not that wordy.  If you were charged at SMS rates for a day's worth of casual web browsing, images and all, you'd probably have to take out a mortgage just to pay for it.  Phone companies have been able to sustain the myth that SMS data is somehow special and deserves to be treated as sacred and precious, fully 1000 times more expensive than the regular bytes you get off the internet, even at the obscene prices they charge for usage-based data plans.

SMS is particularly egregious, but voice isn't that much different.  Phone companies charge such ridiculous rates for "voice" data that Skype built an entire profitable business around giving people the same service for free, and only making money by piggybacking on the phone companies' greed and charging you when sending voice messages over phone networks rather than the internet.  I can't imagine casting the wasteful overhead of legacy phone networks in any sharper relief.

So, we're not there yet, but the market pressure is tremendous to treat data as data, regardless whether it's voice, or SMS, or IM, or "internet" (in other words: everything else, including voice and SMS and IM messages which are sent via different mechanisms).  Until the advent of the recent crop of smartphones, it was difficult and expensive to get an unlimited data plan.  Now, unlimited data plans are the norm, except for "tethering" - using your phone as a proxy for your laptop.  The phone companies are still desperate to convince you that you should pay $60 per month for the privilege of having a USB dongle that you can plug into your laptop rather than just using the mobile IP endpoint — which, by the way, probably aleady has a USB port — that's already in your pocket.

The "mass market" user might not care about operating systems or APIs, but they do understand that a bill with seventeen different break-out metered sections is a bald-faced attempt to rip them off, and a flat-rate or easy to understand pay-as-you-go plan with one number on it is better.

To the extent that phones are not yet interchangeable, unrestricted mobile IP endpoints, it is due to the high barrier to entry to telecom providers, lack of regulation of misleading pricing schemes, and the symbiotic relationship between government and the telecom industry.  However, if one wireless carrier moves to provide simpler billing with more features, the others are forced to follow suit - even more so than cable companies and land-line providers, who can hold their customers hostage via development deals with local governments.  So, this progression is happening, albeit slowly.  For example, when AT&T introduced its iPhone plans, many of the other metered PDA and Blackberry plans, both on AT&T and other providers, began receding from their marketing materials.

Fifteen years ago ... ugh, I feel old.  Let's say ... ten years ago, my computer was barely powerful enough to dedicate all of its processing power to playing one low-resolution movie that took up maybe half the screen.  I was still paying for internet over a phone line with a cap on the number of hours I could use it.  Today, I have real-time two-way video connection to anywhere in the world, 24-7, for a single flat rate.  I own a device that fits in the palm of my hand which contains days worth of continuous music, a library of dozens of books, and connects to the internet.

So, back to that "mass market consumer".  Maybe they don't care about my Python console or IRC chat or SSH access applications, but most "mass market" people do listen to music and read books.  And they're going to care about those features being on their phones, and remaining cheap enough that they can use those features without worrying that they'll go broke if they feel like changing out their playlist.  Also - nobody is really a "mass market" consumer, anyway.  You might not be technical, but maybe you're a golfer, or a swimmer, or a finance nerd.  You want to be able to check the weather on your mobile, or update your latest personal best lap time, or get updates when stocks hit certain price threshholds.  Nobody cares what APIs these apps use, or even whether you call them "apps", but everybody has one extra thing they'd like their mobile to do.

The increasingly ubiquitous, user-customizable, network connected, commodity pocket computer is exactly the technology that is going to deliver that.  It's going to have to become commoditized, which means it's going to be standardized, and secured, which means it's not going to be locked up in carrier notions of what's a "text message" and what's a "voice call" and allow for precise price segregation of every different type of data.

In the future, almost every device will be a computer, albeit with specialized peripherals to assist with performing tasks.  If we're lucky, they will be networked together in standard ways to allow us to control all of them in a consistent and convenient way.

This progression towards computers is good for all of us.  Trust the computer.  The computer is your friend.

Who Wants To Know?

Alternate Titles:

I Have No Log File, Yet I Must Scream

or

So You Logged It, Now What?

or

This is why I don't want to have LOG_DEBUG in Twisted

Sometimes, when writing a program, you feel compelled to make the program emit some output which is peripheral to its operation.  The question is - who wants to know about that information?

Maybe you're debugging the program, and you insert a simple 'print' statement to get some information about it.  Maybe your program is a network server, and you are recording the fact that a message was received and processed.  Maybe you're maintaining an old library routine, and you want it to emit a message that points to a newer, better version of that routine which is now preferred.  Finally, regardless of what kind of program you're writing, maybe it has produced an error that a user or administrator will need to deal with, and you would like to show it to them.

This activity is referred to in several different contexts depending on how the messages are delivered, but it is most commonly known as "logging".  It is critical to the operation of many, many different kinds of programs.  Unfortunately, it is one of the most poorly-understood and poorly-implemented areas of software in general.  Software is a veritable cornucopia of poorly-understood and poorly-implemented ideas, so that's really saying something.  You can see some of the more hilarious and visible examples of developers getting this wrong in the "Pop-Up Potpourri" series on the Daily WTF.

It might seem odd that I lump together funny dialog boxes with "logging".  A dialog box is a little square on your screen; a log message is some text in a file somewhere.  But they are very much the same thing, and they fail in very much the same way.  Log files just do it less visibly.

The point that I hope to communicate here is that for every producer of information, there is a consumer.  When most programmers need to produce a "log message", however, they are thinking only of getting the information out of their program in some format, any format; not how that information is going to be used later.

When I say "most programmers", I most definitely include myself.  I'm probably guiltier than most. one of the reasons I'm writing about this in the first place is to work out some better approaches to the problem.

Consider this output from the "tomboy" desktop sticky-notes program on Ubuntu Hardy.  If I start it from the command-line, I see this:

[DEBUG]: NoteManager created with note path "/home/glyph/.tomboy".
[INFO]: Initializing Mono.Addins
[DEBUG]: AddinManager.OnAddinLoaded: Tomboy.Tomboy
[DEBUG]:            Name: Tomboy.Tomboy,0.10
[DEBUG]:     Description:
[DEBUG]:       Namespace: Tomboy
[DEBUG]:         Enabled: True
[DEBUG]:            File: /usr/lib/tomboy/Tomboy.exe
[DEBUG]: Updating note XML to newest format...

It goes on for several hundred more lines just at startup, and continues to produce messages as the program runs.  These messages are diligently classified into categories: DEBUG and INFO.  I'm sure they're useful to someone.  But why am I seeing them?  I just wanted to start a program to put some sticky notes on my desktop, and none of this information is useful to that task.

I have to imagine that pretty much all of these messages are useful only to Tomboy's developers.  But, worse than the fact that I see them is the fact that if something really interesting happened — I discovered a critical bug, let's say — all of that log output which is being splatted onto my screen is going nowhere.  It is a book written on water.  (Well, a book written on video memory, which is pretty much the same thing.)  Meanwhile, thanks to the bug-reporting facilities in Ubuntu, I'm sure that I could opt to give the Tomboy developers a huge ton of mostly useless information, like the contents of my registers at the time that it crashed.

Consider not just the placement of the messages (on my screen, where I certainly don't care about them) but their formatting.  Who is that elaborate right-justification of labels in the "DEBUG" output for, anyway?  It isn't for me, I don't want to see these messages in the first place.  I doubt it helps the developers, either; rather than just grepping for '[DEBUG]: File', now they need to put in a regular expression to collapse whitespace, or count the number of spaces that the justification happens to put in.  Presumably if this output is useful at all, it is useful in a search.

Text Formatting and the Inevitable Descent into Log-Level Hell

The right-aligned pretty-printing is a beautiful illustration of a very common anti-pattern in logging: trying to convey structured information by messing with a textual format.  A developer wanting to write a message indicating that there is a problem with the program, left with the extremely narrow confines of a logging API which just takes a string, will often do something like this:

log("*****")
log("THIS SHOULD NEVER HAPPEN!  HELP!!!")
log("*****")

Of course, this frantic wording doesn't help the output go anywhere but silently into a log file where it will be ignored.  But, perhaps if this is some server software, an administrator will notice this message and set up an alert that makes their blackberry buzz when they notice those particular words show up in the log file so they can ssh in and look for problems.

Then the developer gets chastised by his manager for his un-informative error message, and updates it to be something clearer:

log("Serious Error: phase inducers have been depolarized.  Contact engineering immediately.")

Of course this breaks the administrators' alerts, so after much discussion between programmers and admins, log levels are added so that admins only get alerts when something "really bad" happens, where "really bad" is an agreed upon flag:

log2(SERIOUS_ERROR, "phase inducers have been depolarized.  Contact engineering immediately.")

Okay.  Now we've got a log level so admins can tell when their pagers should go off.  Except, different developers have different ideas about what "serious" means.

log2(SERIOUS_ERROR, "OMG I lost my cat Mittens.  Where is my cat?")

Clearly this is an abuse of the new "severity" flag that was added, but the cat-engineering team thinks that loss of a cat is pretty serious, so we add a new thing, a log "system".

log3(SYSTEM_CATS, SERIOUS_ERROR, "OMG I lost my cat Mittens.  Where is my cat?")

Most logging systems stop in this general vicinity, but we still haven't solved the problem, which is that the log message has no structure and you can't tell what's going on without groveling around in a bunch of text files with regular expressions or manually reading each message.  Which cat was lost?  Which phase inducer was depolarized?  How do we get from a log message or alert to this information?  The 'log levels' solution to this problem is clearly untenable:

logRidiculous(SYSTEM_CATS, ALERT_IF_YOU_LIKE_CATS, O_RLY, YA_RLY, SERIOUS_BUSINESS, BUT_NOT_TOO_SERIOUS, CAT_LEVEL("Mittens"), "OMG I lost my cat Mittens.  Where is my cat?")

More importantly, if you're writing a library, you have a bunch of other problems.  This diagnostic information needs to be logged somewhere, but what if this library is being used on a user's desktop machine?  Some of these messages are relevant to them as well.  How do you tell the user who is using a GUI that a cat has been lost?  How do you show them the picture of Mittens so they will recognize her if they see her?

Everyone agrees that log messages need some "small amount" of information associated with them, but very few people can agree on what that information should be.  Even at the simplest layer, the idea of a "level", there are lots of open questions.  Is the "debug" level for a programmer trying to debug something on their test rig, or is it for administrators trying to debug something in production?  Should there be a difference between those two things?  How serious does a problem have to be before warranting a "critical" classification?

Once you're using logging code written by more than one programmer, or worse yet, more than one team, you're going to be facing this problem.

The Particular Problem of Libraries

This is, of course, my main interest, since this is where the rubber meets the road for Twisted.  Libraries need to communicate to several different audiences:

1. We need to tell developers using the library about the correct way to use the library at runtime.
   
2. We need to tell administrators of systems using the library about the status of the library and tasks they may need to perform to keep it functioning well.  (Clear your caches, restart the server, install a security update...)  We also need to provide administrators with information they can mine for statistics about how the library is performing; how many requests handled, where its resources are going, etc.
   
3. We need to notify users of applications using the library about things that the library is doing which may be relevant to them.  (A new message has arrived, a new printer is available... obviously this depends heavily on what the library does.)

Libraries, especially event-driven engines such as Twisted, libevent, and glib, have a particularly difficult time because they have to deal with all of these audiences simultaneously.  However, I think that any application or server which needs to do some kind of logging or user notification needs a subset of these features, so if any logging system could solve this problem, it could solve pretty much all logging problems.

Type of Information by Type of Audience

Developers, Developers, Developers

Many languages don't have a solution to developer communication at all.  Python has one — the warnings module — but it is in many ways inadequate.

The warnings module doesn't easily let you selectively see which libraries you want to see warnings for.  If I'm developing an application A using libraries M, N, and O, which themselves have dependencies on X, Y, and Z respectively, I don't want to see warnings that M caused in X or that O caused in Z; those are problems for the maintainers of M and O.

I am maintaining only A, so I want to see warnings caused by my application in M, N, and O.  I can try to filter specifically by module, but unfortunately the
only way of determining which library caused which issue is by directly
examining stack depth, which is unreliable at best and misleading at
worst.  Even if I could filter very accurately, it's hard to get a stand-alone report of warnings and deal with them as they're supposed to be.  Warnings show up to end-users as well, and to administrators looking at applications in production.  It's worth putting up with that to have at least some solution for communication with developers, but it would certainly be better if it didn't happen.

Finally, it's easy to generate a huge amount of warning noise (and, especially as of Python 2.6, many libraries do).  With that much noise and no reporting functionality it's hard to the warnings you care about.

A better solution for communicating for developers would be one which:

• allowed developers to declare somewhere what code they are working on and what code they are just using
• recorded relevant warnings to a log file which was optimized, perhaps with an associated tool, for locating and removing the sources of the warnings
  
• allowed end-users to easily communicate their warning data to developers without inundating them with irrelevant noise while using the application
  

Administrators

To communicate with administrators there is a huge variety of options, but many of them depend on a lot of ad-hoc hackery by the admins themselves, which means they are inconsistent and therefore there is little reusable technology or standardized APIs available.

Right now the gold standard for talking to admins seems to be just writing strings into a text file and hoping they have some facility to read it.

A better solution for communicating with administrators would be one which:

• preserved structured data in an analysis-friendly format, rather than formatting it in human readable messages. (For most UNIX admins, I imagine some kind of structured text would be best, so "grep" would still work but more advanced tools could also be brought to bear.  I'm not sure what the tools in the Windows world look like.  The "Event Viewer" looks like maybe it's a step in the right direction, but its UI is incredibly primitive.)
• provided easily-accessible hooks for dispatching different types of events to ad-hoc code to wire up to existing notification systems - without significantly altering the behavior of the system doing the logging, if the logging hooks were broken, as admin-written code tends to be a bit flaky
  
• included an enumerated list of events which administrators could inspect before they happened to run across them in log files

Although it's a crappy format in many ways, the Common Log Format for HTTP might serve as a good example.  Unfortunately it's too purpose-specific to extend to do more than what it already does, but lots of tools have been written to produce lots of interesting data from even that very simple standard.

Desktop Users

There are two popluar cases for communicating with end-users.  One case is that you're actually running a program on their desktop and you want to tell them something.  Another is that you've got some code running in a web application which wants to tell them something "out of bounds".

On the desktop, there are fairly standard "notification" APIs for popping up little bubbles.  On the web, there are emerging conventions for these notifications, like a bar that descends from the top of the page to mimic the firefox 'do you want to remember this password?' UI element.  A good example of this is Stack Overflow's notification banner.

Unfortunately both of these have a problem with scale and with timing.  If your application suddenly encounters a large number of errors, it will flood the user's screen.  If the user isn't present when a notification occurs (or navigates away), the bubbles or banners may disappear.

A better way to talk to end-users would be:

• for desktop applications, a mini-email interface, which records notifications in a scrolling list so that users can inspect notifications that occur while they're away.
  
• for web applications, a standard API so that multiple applications on a single site (or even, potentially, on different sites) can drop notifications into a queue which can be displayed appropriately.  (Since websites tend to have strong preferences to control their own design, an actual standard widget might not be possible, but it would be nice.)
• for both of these, a standard protocol which would enable notifications to be easily streamed to different computers or mobile devices without needing to reconfigure
  
• a specific classification of messages at the API level, saying, "I want to tell the end-user about this".  Messages about crashes, etc, should be displayed as an option to send information to the developer.  In the context of a web application this can be done automatically and silently; in a desktop app there would need to be some channel set up for sending that information.
  

Let's not forget that administrators are users too.  Everything that happens in a server's notification APIs should be able to be trivially filtered and redirected to administrator's desktop machine (or their phone) so they can immediately notice when something has gone wrong.

Appendix A: Optimization and Dynamic Instrumentation

This doesn't fit into the "figure out what you need to say and who you need to say it to" theme of the rest of the requirements here, but it is nevertheless important.  If you make heavy use of a logging system, especially one where you have lots of messages that are logged "just in case" and rarely displayed, you will quickly discover that it's consuming a lot of resources.

The work to calculate what goes into a log message shouldn't be done unless the message is actually going to be emitted.  Similarly, it should be easy to dynamically add and remove log events from particular methods or particular objects without modifying their code.  The best way to do this is to keep the logging entirely separate and add it at the level of methods and functions, rather than ad-hoc in the middle of application code.  Sometimes, of course, you want to emit log messages at a very particular point, but in general it should be easy to add instrumentation to a method without modifying its code, to avoid cluttering up application logic with lots of "just in case" debug messages.

What Can Do This?

I'm not aware of any logging system that can already do these things.  We'd have to write a new one.  This essay was largely composed due to my desire to understand what I thought a "good" logging system would do.  It might be too ambitious.

There are a few fundamental units which are missing from most logging systems.  While lots of logging systems have various ways to indicate subtle and nuanced levels of urgency, few have a way to indicate who a message might be relevant to.  Logging systems also generally don't have a good way to associate structured information along with a log message.

Twisted's logging mechanism does have a free-form dictionary associated with each message, but that's not much help unless you impose your own structure on it, which means you have to build all this infrastructure anyway.  It is, at least, possible to treat the Twisted system as a kernel which this could be based upon.

In order to produce an enumeration of events before the events are actually logged, this API will require pre-declaration of log messages.  This might be too much of a burden, since sometimes you want to just stick a log message into the middle of some code so that you can see if it happens.  So, in practice, it's more likely that pre-declaration will need to be optional and you'll need to be able to associate ad-hoc data and have it still be persisted along with your message.

There will need to be some way for communicating both the structured data of an event and the human-readable text associated with that event — preferably in a way which can be internationalized.

There's also a bunch of UI, web, and protocol standardization work that would need to be done.  Luckly, that's independent of the actual log machinery; if it already existed it would be a trivial matter to hook it up.  In the meanwhile, something that did all of this but just used existing facilities, like the desktop notification spec, status icons and email, would still be immensely useful.

Sponsor Twisted!

I've kicked off our 2009 sponsorship drive with a post on the Twisted Matrix Labs blog.

Please share and enjoy.

Making easy_install work with Combinator

This is posted mainly for my own benefit, so that I won't have to re-remember the command-line options to make this work for the nth time in a row, but some of you may enjoy it:

    easy_install --prefix ~/.local/ --site-dirs ~/.local/lib/python2.5/site-packages your_package_here

Now that I've gone through and understood why that's necessary, I think I might be able to fix it in Combinator one day...

Notification Disappointment in Ubuntu Jaunty

I have recently been working on some applications that make use of the features provided by notification-daemon.  I installed Jaunty to check out how the much-vaunted new notifications framework works and how it would affect these applications.  I was aware that it sacrificed some features, but I wasn't worried.  I am generally a fan of the GNOME philosophy of dropping functionality and configurability when it doesn't really serve the user.  I also appreciated the new, more minimal and sleek-looking graphic design of the notifications.

I feel like I need to preface my reactions with an apology.  I really like Ubuntu.  In many ways Jaunty looks really slick, and I'm generally enthusiastic to upgrade.

But this notifications stuff?  Wow.  What a disaster.

First, a little background.  I recently started working on two different applications which made heavy use of the notification API. While I originally thought I'd need to implement some of the notification features that I wanted on my own, I was pleasantly surprised to discover that notification-daemon provided almost all of them.

I want to present the user with a time-critical notification, one which didn't grab the focus and interrupt their work.  I want to show the user how much time they had to respond, and provide a few different options for responding to certain notifications.  In one application, I'm implementing a sort of dead-man's switch, where failing to respond within the time limit also counts as a negative response.  I also want to emphasize certain notifications, and provide hyperlinks to their web-based origin so the user can jump straight into the application at the appropriate point if a notification is interesting.

Some of the notifications I want to generate are notifications of events, some are indications of a change in status of my application; so sometimes I want to point at a particular status icon and sometimes I wanted to just drop the notification into a queue - hopefully a global queue which would intermix with other notifications.

I was assuming that I'd need to implement some of these features myself, but to my pleasant surprise notification-daemon handled every single use-case, more or less exactly how I envisioned it working.  I was thrilled.  As part of searching around for "notification" stuff, I learned that Jaunty will have some newer, even cooler notifications stuff, and I was really excited.  Unfortunately, Notify-OSD, the new Ubuntu-specific notification daemon, drops nearly all of these features.  So, I'm back to square one.

Here are some of the specific things which bothered me about it.

1. Applications which emit a notification that prompts for an action ­— something which they only would have done if they explicitly wanted to avoid grabbing focus, since popping up a dialog box is easy enough — will have a modal dialog box pop up and grab the user's focus while they're working.
   
2. Timeouts are no longer honored.  It so happens that in my application I have an operation which takes 2 minutes to time out; I would really like my notification to stay on-screen for this entire time.  I can still do this with notify-osd, but in order to do so I have to watch for the "closed" event and constantly create new notifications.  A smoothly animating timer was a much nicer interface than a sequence of bubbles saying "In 30 seconds I will time out.  In 25 seconds I will time out.  In 20 seconds I will time out."  Yes, I realize that the Ubuntu desktop team will say that I should do something different in this situation, but they're not designing my application and I don't like the options they've suggested.  This would be much less of a problem if the provided notification timeout weren't so distressingly fast.  As a user, by the time I've realized that a notification has popped up and taken the time to focus my eyes on it, it disappears halfway through my reading its message.  I am a very fast reader, and I'm pretty sure that there are a lot of people who are never really going to notice any notify-osd bubbles; they're so fleeting, they're just visual noise.
   
3. Markup is now silently ignored.  I can't emphasize portions of a notification with a larger font, or provide a hyperlink to the origin of the notification.  Similarly, since actions have been broken, I can't provide an action to jump to what caused the notification.  Notifications have thus become disconnected UI element which tell user something while providing them absolutely no tools to deal with it or respond to it.  For example, when Pidgin tells me that a user has signed on, I can no longer interact with the notification to say "yes, I would like to talk to that person".  I have to switch windows to the buddy list, locate the person who just signed on, and click on their name, rather than just clicking on the notification itself.  Or, I have to click on the tiny notification-daemon status icon that's a hard target to hit with my mouse, rather than a big friendly button.
   
4. Notifications are now displayed in the upper-right-hand corner of the screen (where important chrome, like close boxes, search boxes, toolbars, and menus frequently reside) rather than in the lower-right, where less important application features are traditionally located.  Granted, this is a damned-if-you-do-damned-if-you-don't situation, because some applications put important action buttons in the lower right, but I have been learning where to position my windows to avoid that area of the screen for years, and now I have to learn new habits.
   
5. Notifications can no longer be positioned on screen, relative to either widgets in windows or status icons.  This removes the ability for an application to use notifications to draw attention to a particular area of the screen.  Instead, users must make some connection between the notification bubble and the status icon themselves, perhaps by identifying some common graphical element.  If you're already using the "icon" area of the notification to display a picture (such as a person's portrait) it's a bit cramped to also show a copy of the status icon, especially given that the icon will now be squished for you so it's hard to get a pixel-accurate rendering of the status icon anyway.
6. Some of these problems are nominally addressed by the new "indicator applet" facility in Jaunty, but...
   
   1. The indicator applet and libindicate library appear to be almost completely undocumented; the "reference manual" looks like it was an auto-generated stub.  The automatically generated API documentation isn't hosted online anywhere.
   2. The python bindings for libindicate are similarly undocumented, and they aren't packaged anywhere, not even a PPA.  They also use autoconf, rather than distutils, for installation, so their build process doesn't produce a usable extension module and thus they resist installation anywhere but in /usr.
      
   3. I tried to read what passes for documentation — the patch to Pidgin's libnotify plugin that switches it to use libindicate for some things.  I tried it out because I wanted to see if maybe the indicator applet could address some of my concerns, but my misuse of the API caused the indicator applet to instantly segfault.
   4. From what I can tell, you can't just provide an icon and some text, you need to actually create a .desktop file, which means that packaging applications which want to use the indicator applet automatically gets two additional layers of complexity: first, you need to create a .desktop entry, and second, you need to figure out a way to have your application include it during installation.
      

It's interesting to read the version history for Growl, the OS X notification tool which so clearly provided the visual inspiration for Notify-OSD, and notice that many of the features now being removed (application level positioning, close buttons on notifications) are features which were added to later versions of Growl.

The "notification design guidelines" provide some very vague suggestions for ad-hoc mechanisms to work around these regressions.  These suggestions are unhelpful.  I want an API that I can call, not a picture of a window I need to re-create myself.  If the desktop team wants to change the look of my application in the future, I don't want them to submit a giant pile of patches against it.

Not only are the suggestions not a library, they don't include sample code, either.  How do I actually create an alert box which doesn't take focus, doesn't include any window manager controls, but stays above other windows?  Describing it this way is an invitation for applications to be inconsistent.  My interpretation of this specification will inevitably be different from other application authors.  For example, the specification doesn't say anything about compositing, but the window in the screenshot clearly appears to be alpha blended with the background.  It also doesn't say anything about WM controls, but some pictures have minimize/close buttons and some don't.  Some applications will have alpha blending for their notifications, some won't.  And since there's no library here, there's no reasonable way to consistently control the behavior of many applications.

With these features in libnotify, we have a single uniform queue for users' attention.  A single point of control which might be adjusted, bugfixed, and tweaked across the desktop as a whole, without writing tons of patches for individual applications.  Notify-OSD even takes advantage of that point of control.  But the suggestions for many of these features is to take control out of a single easily-managed client/server protocol and push it into a bunch of ad-hoc application-specific widgetry.

To add insult to injury, the one place that I do get hard examples of how to do things, on the notification development guidelines page, the Python code samples have yet to be written.  This is a minor nit, as I can definitely figure out what's going on from the C# code, but it's endemic of the same systemic problem with the Linux desktop ecosystem that brought us this half-baked replacement for notification-daemon.  Jamie Zawinski identified this problem as the "Cascade of Attention-Deficit Teenagers", but Canonical demonstrates that you can create this same problem with a medium-size company that employs highly competent, adult engineers.

Notification-daemon isn't perfect.  It could clearly stand to be improved, especially in the face of notification spam.  So please, improve it!  Or, at worst, if upstream is not cooperative, fork it.  I'm pretty sure that the solution to the rate limiting problem is not "then... what?".  Notify-OSD cuts the gordian knot of notify-spam by only letting you see one thing at a time, but that has its own problems.

Now, to be fair, all these regressions haven't really cost me any work.  I want my applications to be cross-platform, so I was going to have to implement most of this functionality for Windows anyway, using animating borderless windows.  Now I'm just going to be using my own notification widgetry on Ubuntu as well, rather than elegantly integrating with the platform and providing all of my notification interaction through a familiar UI.  But I'm sad that the superior notification infrastructure on Linux in general and Ubuntu specifically is no longer something that makes my application easier to write first.

So, beyond this one little screed, I'm really not going to complain too much.  I'll implement some of my own ideas for notification, try to come up with some way to be friendly to Notify-OSD in the meanwhile, and I'll still eventually upgrade all of my computers to jaunty and enjoy the other eye-candy and performance improvements.  This is not too terrible of a price to pay, and I do keep it in perspective.  I also understand that I'm not the guy who has to make the hard decisions for what goes into Ubuntu or GNOME or whatever.  I understand that sometimes, in order to make an omelette, you have to kill a few people.

But, if a decision maker for Ubuntu were to care about my entirely irrelevant opinion, as both an application developer and heavy user of notification systems, I would say this to them:

You guys have done some great work on Notify-OSD.  It's a worthy prototype.  In many ways it is better than notification-daemon: it looks nicer, it makes notifications between applications more consistent.  I can appreciate the uncompromising vision you have for cleaning up the sometimes confusing pile of notifications that users see.

You should package Notify-OSD in Jaunty, so that people start using it.  Start updating applications to honor the capabilities that it provides.  But please, don't make it the default in the first release where it's included, and yes, include a preference for the period of transition.  Write some libraries to support the other use-cases which Notify-OSD right now ignores.  Document and stabilize the indicator applet.  Package the Python bindings, please.  Make it not crash when applications abuse it.

Regardless of this new notification system's unpolished state, I'm sure many users will update and start experimenting with Notify-OSD, much as many started with Compiz for years before it became the default window manager.  Most users can keep using the regular notification bubbles until Karmic, though.  When Karmic comes along, you'll have had the time you need to finish the documentation and provide application developers better alternatives to a good notification API before yanking the carpet out from under them.

If this advice is ignored, as I'm almost sure it will be, it won't bother me - notification is hardly the most important API that an OS provides.  The thing that I really hope someone will take away from this is the general theme that platforms should evolve experimental features slowly, and you should always have a well-documented, better alternative ready before you remove something.  Notify-OSD removes a half-dozen features and informally, halfheartedly gestures at some ways you can make your window pop up to address what some of those features used to do.  That's fine, as a scrappy new competitor to notification-daemon, but not as a core part of a major platform.

My real hope is not specifically that Notify-OSD will actually be pulled.  Of course I'd be happy if it were, but Jaunty going to be released in just a few days.  Again, I feel like I need to qualify these statements: if I'd really wanted to impact decisions like this I should really be regularly using beta releases.  Not to mention the fact that if I'd actually finished these hypothetical applications I'm thinking about, I'm sure my voice would carry more weight.

My real hope is that you, gentle reader, will take this message, and the next time you are contemplating boiling your favorite ocean, you'll stop and reflect.  Break down the changes you are planning on into individual, incremental improvements, rather than sweeping, break-everything lateral movement.  Make radical improvements, but make them behind the stable facade of a system which is only lifted when the radical improvement is clearly both radical and an improvement.

My Time At PyCon

I tried to write a conference wrap-up, but there's just too much.  Besides, Ted Leung already wrote a better one than I could.  (I'm pleased to note that the first talk he mentions is the only Twisted talk at the conference: "Twisted AMQP and Thrift".  Go Esteve!)

I am once again impressed by the conference organizers.  Every year it seems like the conference gets better.  This year, it seemed like nothing went wrong - which wouldn't be remarkable if I didn't know just how difficult it is to create that impression.  I registered for the conference online, reserved my hotel room through the housing committee, and when I showed up everything was ready.  The food was good, all of the staff were helpful and efficient.  There was plenty of room for everything, even though the Open Space board was completely packed.

The video was particularly awesome.  I didn't get to see many talks, as I was moving from one conversation to another in the hallways, but I was impressed to discover that some of the talks I was hearing about during the conference were already online so I could catch up on the buzz.  The camera work and editing are really good.  I particularly liked that they often used picture-in-picture to show both the presenter and their slide.  But, you don't have to take my word for it; check them out now on pycon.blip.tv.

My personal experience of the conference is of course defined by hallway chats, open spaces and sprints.  This year we had a great Twisted open space, followed by a great Twisted Web open space the next day.  Thank you to everyone who came.

These sessions helped to reinforce for me the need to repeat this frequently for our users: if Twisted is doing something which confuses you but seems wrong, please go ahead and file a bug on twistedmatrix.com.  If you're wrong, and Twisted is working properly, there's still a bug, it's just a bug in the documentation.  If the documentation were perfect presumably you would understand why it's doing what it's doing.  Even if we ultimately decide that the documentation is sufficient, by adding a bug in a tracker, you've told Google that people with your question should find the document we refer to in our comments, so there's still value.

The sprints also went really smoothly this year.  Power and wifi were in abundance, so we could all just get to work.  I'll tell you a secret, though: I never plan to get much work done at the PyCon sprint, especially with monthly Twisted sprints here in Boston.  The value in this nation-wide gathering is in helping new users get up to speed with hacking on Twisted, and in running around talking to people sprinting on other projects, getting them to integrate with Twisted.  This year I had a pretty focused message that I wanted to get out: Twisted is a WSGI container, and it's one you can invoke from the command line with "twistd web --wsgi=your.application.here".  In other words, even if you prefer to write synchronous, blocking code, you can still use Twisted to run your web application, and you don't need to write any additional code to do it.  As a result of this communication, David Reid fixed a few minor bugs in our WSGI container and Twisted trunk now runs Django as well as Pinax.  I hope that this will drive even more adoption of Twisted in the Python world.

Also, it seems like for the last few years I've gotten started thinking about PyCon talks too late, and by the time the deadline rolls around I've got nothing.  In fact it seems like this has happened to most Twisted devs over the last few years.  This year I plan to get started right now.  I hope you'll join me, so that we can have a "Twisted comeback tour" at PyCon 2010.

One last thing: at least three people approached me at various points during the conference to ask me about using Mantissa and Axiom, and I didn't have a chance to catch up with any of them.  I feel bad about this.  If you tried to talk to me about using some Divmod technology at pycon, and still want to talk about it, please feel free to send me a personal email; we can set up a Skype session or something.

Tell Everyone You Know

A friend of a friend has had some trouble with the thugs who run the recording industry.

Whatever you feel about file-sharing, I hope we can all agree that it's wrong to legally attack, and bankrupt, a poor college student on a full scholarship, for copying four songs.

While I would, of course, love you all to donate to Twisted, if you only have fifty dollars to donate this year, give it to Fabiola.  We'll muddle through.

Doing Stuff at PyCon

As bloggers are often wont to observe, the best thing about PyCon is the "hallway track".  This is especially true for old hands like me who are already familiar with most of the projects and techniques being introduced and described in the scheduled talks.

Usually I just wander around in the hallways and let random encounters occur as they will.  That's worked out reasonably well in the past, but I think I can get more out of the conference with a little preparation.  In particular, I'd like to use the open-space rooms to do some of this discussion so that we can really come to some conclusions rather than standing awkwardly in hallways, seeming as if we're about to leave.  So, with your help, I'd like to plan things out a little more.  Here are the things that I'd really like to do.

I am going to dedicate the sprint to thinking and talking about development on Twisted itself.  Of course I'm up for having a few meals during the conference with the Twisted developers who I don't get to see in person too often, but I talk to you guys all the time — I'd rather spend the conference talking to people I wouldn't otherwise see.

During the conference, I'd like to try to talk to people who are working on other projects.  Of particular interest to me are PyGame and Django.  There are currently no good, well-documented answers to "how do you use Twisted to make a networked game with PyGame" or "how do you use Twisted to network non-HTTP protocols with an existing Django app".  Both of those seem to be increasingly common questions.  If you are a leader of either of those projects, or can help me schmooze with one of them, I'd appreciate it.  I am, of course, just as interested in hearing from other projects that I don't even know about their twisted-integration stories.

In general I'd like to encourage other projects unrelated to Twisted to do this kind of cross-project jam session if you can.  There's a great opportunity for cross-pollenation at PyCon.

I'd also like to hear from users of Twisted, Nevow, Mantissa, et cetera, especially those who have not been particularly vocal on the mailing lists or IRC.  The best kind of user is a prospective volunteer, of course (stay for the sprints!), but both praise and constructive criticism are also welcome.

Don't underestimate the value of cheerleading!  Open source projects thrive on donated labor, and the fuel for donated labor is enthusiasm.  I can say for myself that a few kind comments at prior PyCons have motivated lots of hacking.  I don't just mean for me, though: take the opportunity to thank your favorite open source Python hacker, whoever that may be.

I'll be raising funds for the Twisted project.  If you're considering sponsorship, I would like to talk to you about how it will make you and everyone in your organization healthier, smarter, and better-looking.  (Mind-control powers and teleportation are reserved for sponsors at the gold level and higher, and only while supplies last.)

Finally, I'd like to talk about some of Divmod's less well-known projects, like Imaginary and Vertex, especially with people who are interested in helping to work on such things.  I am thinking about maybe doing a lightning talk or open space session, if it seems like there is enough interest to justify the preparation effort.

If any of these things are interesting to you, please leave a comment or drop me a note.  I doubt that I'm popular enough, or organized enough, to really do this properly, but I'd like to have at least a vague calendar of pre-planned meetings and open space sessions on my phone so I can avoid wasting any time once I'm actually there.

The Perfect Iced Latte

The only thing I take more seriously than my keyboards is my coffee.  I've been tinkering with my recipe for a while now, but I'm really very happy with the current iteration.  Here's exactly how I make my morning coffee.

1. Grind up roughly ¾ cup of 49th Parallel Epic Espresso using a Krups Fast Touch Coffee Grinder (Black).
2. Filter some water using a Brita Riviera water filter.
   
3. Place the grounds into an IKEA KAFFE french press, and fill it with filtered water.  Depress the plunger to make sure the grounds are soaked with water, then remove the plunger and cover with tinfoil.
4. Put the water into the refridgerator for between 12 and 24 hours.  (More time produces stronger coffee, but after 24 hours it starts to get unpleasantly bitter, and after 72 it's undrinkable.)
   
5. Put the plunger back in, and filter the espresso concentrate.  Decant it into another container.  (This concentrate will last for a week or two.)
6. To serve, mix 1 part concentrate with 3 parts Stonyfield Farm Organic Milk and 1 teaspoon of Butternut Mountain Farm Pure Vermont Maple Syrup.

Disclosure: nobody actually paid me to mention all those products.  I just like hyperlinks.

It's Always Sunny in Python

Jonathan Lange suggests that you have three options when you have some code that needs testing.

1. Give up
2. Work hard to write the damn test
3. Make your code testable.

In fact he starts with just the first two options and then reveals the third, but I suggest that there is an option number four, available only in Python (and other, similar dynamic languages):

4. Cheat.

I agree with Jonathan that you should generally make your code more flexible, and thereby testable.  Any code which is friendly to being invoked as a library or as an independent unit of functionality can be tested, so you should endeavor to never write code that is unfriendly to being executed as just a normal method call.  This is especially true if you are writing new code in a properly test-driven manner.  Keeping in mind, of course, that one should not modify the SUT purely for the purpose of testing.

However, if one is (as I often find myself) adding test coverage to a grotty old system, written at a time or in a place where test-driven development was not the norm, one typically wants to establish test coverage before making any changes to the code or its design.  In such a situation, one may often find oneself in the undesirable position of needing to carefully modify some implementation code so that it can be tested, hoping that none of its untested interactions with other areas of the system will be broken as a result.  For example, you might encounter some paranoid and misguided Java code like this:

    // Startup.java
    private static final void emitLogMessage(final String message) {
        System.out.println(message);
    }

    public static final void startUp() {
        // ...
        emitLogMessage("Starting up!");
        // ...
    }

In this case, it's very difficult to get in the way of any part of this system.  Nothing is parameterized, everything is global, and the compiler won't even let you call one of these methods.  You really only have Jonathan's three options here, none of which are desirable.

1. You can give up on testing this part of the system until you've covered other parts of the system.  In many cases this is the right thing to do, but it is often the lowest-level and most critical parts of a system which have calcified into this sort of untestable rubble.
2. You can work hard to write the damn test.  There are a number of extremely subtle nuances of the Java runtime which you can take advantage of to make a lie of the "private" and "final" keywords.  You can load the code using a custom classloader, manipulate its bytecode, or invoke private methods using reflection.  This is ultimately the "right thing" to do, but it requires the development of a daunting skill-set which you would not otherwise need.
3. You can make the code testable, changing it before you've properly tested the code which is already in use.  Ultimately this is what you want to get to anyway, but if the code is doing something subtle that you didn't test (and none of the rest of the system is tested yet) you might be (rightly) concerned that this could break something else.

In Python, the hard work is not so hard.  To start with, Python doesn't have the misfeatures of private and final.  It also doesn't have any baroque "reflection" constructs.  All you need to understand is attribute access.  So, if you have a similar Python file:

# startup.py
import sys

def emitLogMessage(message):
    sys.stdout.write("%s\n" % (message,))

def startUp():
    # ...
    emitLogMessage("Starting up!")
    # ...

Idiomatically, the situation looks just as hopeless.  Everything is global, and nothing is parameterized.  It's hard-coded.  However, if you look at it from the right angle, you will realize that you can't really code that "hard" in python.

What emitLogMessage is doing in this case is not making a fixed reference to the global sys module: it is simply accessing the sys attribute of the startup module.  So in fact, we can easily test it:

# test_startup.py
import sys
import startup
import unittest

class FakeSys(object):
    def __init__(self, test):
        self.test = test
    @property
    def stdout(self):
        return self
    def write(self, message):
        self.test.messages.append(message)

class StartupTest(unittest.TestCase):
    def setUp(self):
        self.messages = []
        startup.sys = FakeSys(self)

    def tearDown(self):
        startup.sys = sys

    def test_startupLogMessage(self):
        startup.startUp()
        self.assertEquals(self.messages, ["Starting up!\n"])

So testing startUp is a simple matter of replacing the sys object that it's talking to: which, in Python, is rarely more than a setattr() away.

I've taken care here to use only standard Python features.  This is, after all, theoretically possible in Java, it's just a heck of a lot harder, both to use and to understand — the learning curve is a big part of the problem.  However, if you're using Twisted, and willing to spend just a brief moment to learn about one of its testing features, you can save a few lines of code and opportunities for error:

import startup
from twisted.trial import unittest

class FakeSys(object):
    def __init__(self, test):
        self.test = test
    @property
    def stdout(self):
        return self
    def write(self, message):
        self.test.messages.append(message)

class StartupTest(unittest.TestCase):
    def setUp(self):
        self.messages = []
        self.patch(startup, "sys", FakeSys(self))

    def test_startupLogMessage(self):
        startup.startUp()
        self.assertEquals(self.messages, ["Starting up!\n"])

Please keep in mind that this is still not the best way to do things.  Use the front door first.  It's much better to use a stable, documented, supported API in your tests than to depend on an accident of implementation which should be able to change.  However, it is even worse to associate the feeling of testing with the feeling of being stuck, being unable to figure out how to dig yourself out of some hole that old, bad design has dug you into.

I'm writing this mostly for people who are new to test-driven development in Python and think that unit tests need to be a huge amount of extra work.  They don't.  If you ever find yourself struggling, unable to figure out how you could possibly write a test which would exercise some tangle of poorly-designed code, just remember: it's all just objects and methods, attributes and values.  You can replace anything with anything else with a pretty trivial amount of effort.  Of course you should try to figure out how to improve your design, but you should never think that you need to stop writing tests just because you used a global variable and you can't figure out what to replace it with.

Explaining Why Interfaces Are Great

Why?

Why use interfaces? Especially with Python's new ABCs, is there really a use for them?

Some of us Zope Interface fans — names withheld to protect the guilty, although you may feel free to unmask yourselves in the comments if you like — have expressed frustration that ABCs beat out interfaces for inclusion of the standard library.  However, I recently explored various mailing lists, Interfaces literature, and blogs, and haven't found a coherent description of why one would prefer interfaces over ABCs.  It's no surprise that Zope's interface package is poorly understood, given that nobody has even explained it!  In fact, PEP 3119 specifically says:

For now, I'll leave it to proponents of Interfaces to explain why Interfaces are better.

It seems that nobody has taken up the challenge.

I remember Jim Fulton trying to explain this to me many years ago, at a PyCon in Washington DC.  I definitely didn't understand it then.  I was reluctant to replace the crappy little interfaces system in Twisted at the time with something big and complicated-looking.   Luckily the other Twisted committers prevailed upon me, and Zope Interface has saved us from maintaining that misdesigned and semi-functional mess.

During that explanation, I remember that Jim kept saying that interfaces provided a model to "reason about intent".  At the time I didn't understand why you'd want to reason about intent in code.  Wouldn't the docstrings and the implementation specify the intent clearly enough?  Now, I can see exactly what he's talking about and I use the features he was referring to all the time.  I don't know how I'd write large Python programs without them.

Caveat

This isn't a rant against ABCs.  I think ABCs are mostly pretty good, certainly an improvement over what was (or rather, wasn't) there before.  ABCs provide things that Interfaces don't, like the new @abstractmethod and @abstractproperty decorators. Plus, one of the irritating things about using zope.interface is that the metadata about standard objects in zope.interface.common is not hooked up to anything: IMapping.providedBy({}) returns False.  ABCs will provide that metadata in the standard library, making zope.interface that much more useful once it has been upgraded to understand the declarations that the collections and numbers modules provide.

So, on to the main event: what do Zope Interfaces provide which makes them so great?

Clarity

Let's say we have an idea of something called a "vehicle".  We can represent it as one of two things: a real base class (Vehicle), an ABC (AVehicle) or an Interface (IVehicle).

There are a set of operations that interfaces and base-classes share.  We can ask, "is this thing I have a vehicle"?  In the base-class case we spell that 'if isinstance(something, Vehicle)'.  In the interfaces case, we say 'if IVehicle.providedBy(something)'.  We can ask, "will instances of this type be a vehicle?".  For an interface, we say 'if IVehicle.implementedBy(Something)', and for a base class we say 'issubclass(Something, Vehicle)'.  With the new hooks provided by the ABCs in 2.6 and 3.0, these are almost equivalent.  With zope.interface, you can subclass InterfaceClass and write your own providedBy method.  With the ABC system, you subclass type and implement __instancecheck__.

However, there are some questions you can't quite cleanly ask of the ABC system.  For one thing, what does it really mean to be a Vehicle?  If you are looking at AVehicle, you can't tell the difference between implementation details and the specification of the interface.  You can use dir() and ignore a few of the usual suspects — __doc__, __module__, __name__, _abc_negative_cache_version — but what about the quirkier bits?  Metaclasses, inherited attributes, and so on?  There's probably some way to do it, but I certainly can't figure it out quickly enough to include in this article.  In other words, types have two jobs: they might be ABCs, or they might be types, or they might be both, and it's impossible to separate those responsibilities.

With an Interface, this question is a lot easier to ask.  For a quick look, list(IVehicle) will give a complete list of all the attributes expected of a vehicle, as strings.  If you want more detail, IVehicle.namesAndDescriptions() and Method.getSignatureInfo() will oblige.

Since the interface encapsulates only what an object is supposed to be, and no functionality of its own, it's possible for frameworks to inspect them and provide much nicer error messages when objects don't match their expectations.  zope.interface.verifyClass and zope.interface.verifyObject can tell you, both for error-reporting and unit-testing purposes, whether an object looks like a proper vehicle or not, without actually trying to drive it around.

Flexibility

At the most basic level, interfaces are more flexible because they are objects.  ABCs aren't objects, at least in the message-passing smalltalk sense; they are a collection of top-level functions and some rules about how those functions apply to types.  If you want to change the answer to isinstance(), you need to register a type by using ABCMeta.register or overriding __instancecheck__ on a real subclass of type.  If you want to change the answer to providedBy, for example for a unit test, all you need is an object with a providedBy method.

Of course, you can do it "for real" with an InterfaceClass, but you don't need to.  In other words, its semantics are those of a normal method call.

Interfaces aren't completely self-contained, of course: there are top-level functions that operate on interfaces, like verifyObject.  However, there's an interface to describe what is expected:

>>> from zope.interface.interfaces import Interface, IInterface
>>> IInterface.providedBy(Interface)
True

There's also the issue of who implements what.  For example, you might have a plug-in system which requires modules to implement some functionality.  Generally speaking, modules are instances of ModuleType, so specifying that all modules implement some type with an ABC is somewhat awkward.  With an interface, however, there is a specific facility for this: you put a moduleProvides(IVehicle) declaration at the top of your module.

In zope.interface, there is a very clear conceptual break between implements and provides.  A module may provide an interface — i.e. be an object which satisfies that interface at runtime — without there being any object that implements that interface — i.e. is a type whose instances automatically provide it.  This distinction comes in handy when avoiding certain things.  This distinction exists with ABCs; either you "are a subclass of" a type or you "are an instance of" a type, but the language around it is more awkward and vague, especially since you can be a "virtual instance" or "virtual subclass" now as well.

There's also the issue of dynamic proxies.  If you have a wrapper which provides security around another object, or transparent remote access to another object, or records method calls (and so on) the wrapper really wants to say that it provides the interfaces provided by the object it is wrapping, but the wrapper type does not implement those interfaces.  In other words, different instances of your wrapper class will actually provide different interfaces.  With zope.interface you can declare this via the directlyProvides declaration.  With ABCs, this is not generally possible because ABCMeta.register will only work on a type.

Adaptation

Let's say I have an object that provides IVehicle.  I want to display it somehow — and in today's web-centric world, that probably means "I want to generate some HTML".  How do I get from here to there?  ABCs don't provide an answer to that question.  Interfaces don't do that directly either, but they do provide a mechanism which allows you to provide an answer: you can adapt from one interface to another.

I'm not going to get into the intricacies of exactly how adaptation works in zope.interface, since it isn't important to understand most of the time.  Suffice it to say you can adapt based on specific hooks that are registered, based on the type an object is, or based on what interfaces it provides.

The gist of it is that you have some thing that you don't know what it is, and you want an object that provides IHTMLRenderer.  The way you express that intent is:

    renderer = IHTMLRenderer(someObject)

If there are no rules for adapting an object like the one you have passed to an IHTMLRenderer, then you will get an exception - which is all that will happen, normally.  However, this point of separation between the contract that your code expects and the concrete type that your code ends up actually talking to can be very useful.

The larger Zope application server has a rich and complex set of tools for defining which adapter is appropriate in which context, but Twisted has a very simple interface to adaptation.  You simply register an adapter, which is a 1-argument callable that takes an object that conforms to some interface or is an instance of some class, and returns an object that provides another interface.  Here's how you do it:

    from twisted.python.components import registerAdapter
    class VehicleRenderer(object):
        "Render a vehicle as HTML"
        implements(IHTMLRenderer)
        def __init__(self, vehicle):
            self.vehicle = vehicle
        def renderHTML(self):
            return "<h1>A Great Vehicle %s (%s)</h1>" % (
                       self.vehicle.make.name,
                       self.vehicle.model.name)
    registerAdapter(VehicleRenderer, IVehicle, IHTMLRenderer)

Now, whenever you do IHTMLRenderer(someVehicle), you'll get a VehicleRenderer(someVehicle).

Your code for rendering now doesn't need any special-case knowledge about particular types.  It is written to an interface, and it's very easy to figure out which one; it says "IHTMLRenderer" right there.  It's also easy to find implementors of that interface; just grep for "implements.*IHTMLRenderer" or similar.  Or, use pydoctor and look at the "known implementations" section for the interface in question.

Conclusion

In a super-dynamic language like Python, you don't need a system for explicit abstract interfaces.  No compiler is going to shoot you for calling a 'foo' method.  But, formal interface definitions serve many purposes.  They can function as documentation, as a touchstone for code which wants to clearly report programming errors ("warning:  MyWidget claims to implement IWidget, but doesn't implement a 'doWidgetStuff' method"), and a mechanism for indirection when you know what contract your code wants but you don't know what implementation will necessarily satisfy it (adaptation).

Even with a standard library mechanism for doing some of these things, Zope Interface remains a highly useful library, and if you are working on a large Python system you should consider augmenting its organization and documentation with this excellent tool.

Help Us Help You

The Twisted project is getting ready for another round of fund-raising.

Like last year, we'll be centering this effort around PyCon.  This year, we have a year's track-record for our potential sponsors to evaluate us on.

During this year of sponsored development, we closed a record number of tickets this year.  Partially this is due to the work that JP has done himself, partially it is due to the increased rate at which users' contributions have been reviewed.

Aside from raw numbers, funding has allowed us to dedicate the sustained effort required to deal with very old, very unpleasant, very difficult issues like properly handling child process termination and the development of a new, better HTTP client.

But we can't keep this up without help.  Unless you've been living in a very deep hole, you know that the world's economy has exploded and a lot of companies are feeling the pain.  It will be harder in this tough climate to convince companies that this is a good time to invest in software which "doesn't cost anything".

At the same time, we believe that we could get an even better outcome for the Twisted project if we can allocate more funds this year.  We could upgrade from part-time to full-time maintenance, do more new development, and possibly even fund a Twisted conference.

This is where you come in.  The people responsible for raising funds for Twisted are mostly the same people who write code for it.  The more help we can get from you — the developers who use Twisted — the more of our spare time we can spend writing code.

If you are interested in helping with this, especially if you have experience doing fund-raising, please let us know on the mailing list.  This is a great opportunity for those of you who would really like to give something back to Twisted but haven't had the opportunity to contribute code.

(Of course, if you don't have the time to help with fundraising either, you can always make a small personal contribution using the form on the front page of twistedmatrix.com.  Every little bit helps, and donations are tax-deductible.)

I would be remiss if I did not mention that the Software Freedom Conservancy has been extremely helpful in helping us collect donations, manage our accounts, and deal with the legal paperwork of establishing a non-profit.  Without their help we would likely not have had the collective attention span to establish a foundation and lay the foundations that now allows us to collect tax-exempt donations.  If you are contributing to Twisted, please consider contributing something to the SFC as well.

Using SSH Keys on a USB Drive on MacOS X

I keep my SSH private key on a USB thumb drive.

The idea is that I don't want my private key to be on the hard disk of any of the computers that I use.  I use several and so I'm not observing them all constantly, so I don't want to leave my key around for automated attackers to pick it up.

I load the key directly from the USB drive into my SSH agent, which then mlock()s it so it doesn't get put into swap.

This works on Windows (with PuTTY) and Linux just fine.  Unfortunately MacOS X has a nasty habit of mounting FAT volumes with free-for-all permissions, so when I try to load the key:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
         WARNING: UNPROTECTED PRIVATE KEY FILE!          @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0777 for '/Volumes/GRAVITON/....id_rsa' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.

I thought that this was an intractible problem.  The only solution I'd found previously was to make a copy of the key, make a sparse disk image, and manually mount the sparse disk image.  However, this workaround has two problems:

1. It's inconvenient.  I have to manually locate the disk image every time, double click it, etc.
2. It's insecure.  If I ever allow other users to log in to any of my OS X machines, they can read the version of the key I'm not using on the FAT filesystem, even if only I can read the one on the HFS+ disk image.

Today, almost by accident, I discovered the real answer.

The daemon that mounts disk on OS X is called "diskarbitrationd".  I discovered this by running across some OpenDarwin documentation which explains that you can configure this daemon by putting a line into fstab.

First you need a way to identify the device in question.  None of the suggested mechanisms for determining the device UUID worked for me, so I used the device label instead.  This is probably desirable anyway, since at least you can tell when the label changes; if you move your key to a similar device, the UUID is different but you can't tell.

You can set the device label by mounting your USB drive, doing "get info" on it, editing the name in the "name and extension" section, and then hitting enter.  You should use an all-caps name, since when you re-mount the drive it will be all-caps again anyway.

You also need to know your user-ID.  The command 'id -u' will return it.

Then, you need to add a single line to /etc/fstab.  My drive's label is "GRAVITON", and my user-ID is 501, so it looks like this:

LABEL=GRAVITON none msdos -u=501,-m=700

Now, all you have to do is eject your drive and plug it in again.  Voila!

$ ssh-add /Volumes/GRAVITON/....keychain.id_rsa
Identity added: /Volumes/GRAVITON/....keychain.id_rsa (/Volumes/GRAVITON/...keychain.id_rsa)


Now you can securely carry your SSH key with you to macs, without breaking ssh-agent's intended protection.

The Joel Un-test

Joel Spolsky seems to like controversy, although I can see why.  Being a contrarian ideologue is pretty sweet.

Some people have been suggesting that the Joel Test should mention "100% unit test coverage" as well.  Personally, I think that's a great idea.  The industry is evolving, and automated testing is getting into the suite of tools that every competent programmer should be familiar with.

Joel disagrees, since 100% coverage is "a bit too doctrinaire about something you may not need".

For what it's worth, I don't completely disagree with Joel.  Some of the software that I work on doesn't have 100% test coverage, and that's okay.  I wrote it before I learned about unit testing.  I'm not freaking out and spending all of my time just writing tests for old code which apparently works.

However, we do have policies in place to add test coverage whenever we change anything.  Those policies stipulate that 100% coverage is a requirement for any new or changed code, so I consider myself a fan of 100% coverage and I generally think it's a good idea.  I do think it belongs on the Joel Test, or at least something like it.

I feel like my opinions are representative of a pretty substantial number of "agile" practitioners out there, so I'd just like to respond to a few points:

Joel mentions "SOLID principles", as if they're somehow equivalent to unit testing.  As if the sentiment that leads one to consider 100% test coverage a great idea leads one into a slavish architectural death-spiral, where any amount of "principles" are dogma if they have a sticker that says "agile" stuck to them.

Let me be clear.  I think that SOLID, at least as Joel's defined it, is pointlessly restrictive.  (I've never heard about it before.)  As a guy who spends a lot of time implementing complex state machines to parse protocols, I find "a class should have only one reason to change" a gallingly naive fantasy.  Most of the things that Joel says about SOLID are true, especially if you're using a programming language that forces you to declare types all over the place for everything.  (In Python, you get the "open" part of "OCP", and the "clients aren't forced" part of "ISP" for free.)  It does sound, in many ways, like the opposite of "agile".

So, since SOLID and unit testing are completely unrelated, I think we can abandon that part of the argument.  I can't think of anyone I know who likes unit testing would demand slavish adherence to those principles.  I agree that it sounds like it came from "somebody that has not written a lot of code, frankly".

On the other hand, Joel's opinion about unit tests sounds like it comes from someone who has not written a lot of tests, frankly.

He goes on and on about how the real measure of quality is whether your code is providing value to customers, and sure you can use unit tests if that's working for you, but hey, your code probably works anyway.

It's a pretty weaselly argument, and I think he knew it, because he kept saying how he was going to get flamed.  Well, Mr. Spolsky, here at least that prediction has come true ;-).

It's weaselly because any rule on the Joel Test could be subjected to this sort of false equivalence.  For example, let's apply one of his arguments against "100%" unit testing to apply to something that is already on the joel test, version control:

But the real problem with version control as I've discovered is that the type of changes that you tend to make as code evolves tend to sometimes cause conflicts. Sometimes you will make a change to your code that, causes a conflict with someone else's changes. Intentionally. Because you've changed the design of something... you've moved a menu, and now any other developer's changes that relied on that menu being there... the menu is now elsewhere. And so all those files now conflict. And you have to be able to go in and resolve all those conflicts to reflect the new reality of the code.

This sounds really silly to anyone who has really used version control for any length of time.  Sure, sometimes you can get conflicts.  The whole point of a version control system is that you have tools to resolve those conflicts, to record your changes, and so on.

The same applies to unit tests.  You get failures, but you have tools to deal with the failures.  Sure, sometimes you get test failures that you knew about in advance.  Great!  Now, instead of having a vague intuition about what code you've broken intentionally, you actually have some empirical evidence that you've only broken a certain portion of your test suite.  And sure, now you have to delete some old tests and write some new tests.  But, uh... aren't you deleting your old code, and writing some new code?  If you're so concerned about throwing away tests, why aren't you concerned about throwing away the code that the tests are testing?

The reason you don't want to shoot for 90% test coverage is the same reason you don't want to shoot for putting 90% of your code into version control or automating 90% of your build process into one step or putting 90% or (etc) is that you don't know where the bugs are going to crop up in your code.  After all, if we knew where the bugs were, why would we write any tests at all?  We'd just go to where the bugs are and get rid of them!

If you test 90% of your code, inevitably, the bugs will be in the 10% that you didn't test.  If you automate 90% of your build, inevitably the remaining non-automated 10% will cause the most problems.  Let's say getting the optimization options right on one particular C file is really hard.  Wouldn't it be easier to just copy the .o file over from bob's machine every time you need to link the whole system, rather than encoding those options in some kind of big fancy build process, that you'd just have to maintain, and maybe change later?

Joel goes on to make the argument that, if he were writing some software that "really needed" to be bulletproof, he'd write lots of integration tests that exercised the entire system at once to prove that it produced valid output.  That is a valid testing strategy, but it sort of misses the point of "unit" tests.

The point of unit tests — although I'll have to write more on this later, since it's a large and subtle topic — is to verify that your components work as expected before you integrate them.  This is because it's easier to spot bugs the sooner you find them: the same argument Joel makes for writing specs.  And in fact if you read Mr. Spolsky's argument for writing specs, it can very easily be converted into an argument for unit testing:

Why won't people write unit tests? People like Joel Spolksy claim that it's because they're saving time by skipping the test-writing phase. They act as if test-writing was a luxury reserved for NASA space shuttle engineers, or people who work for giant, established insurance companies. Balderdash. ... They write bad code and produce shoddy software, and they threaten their projects by taking giant risks which are completely uncalled for.

You think your simple little function that just splits a URL into four parts is super simple and doesn't need tests because it's never going to have bugs that mysteriously interact with other parts of the system, causing you a week of debugging headaches?  WRONG.  Do you think it was a coincidence that I could find a link to the exact code that Joel mentions?  No, it's not, because any component common enough to make someone think that it's so simple that it couldn't possibly have bugs in it, is also common enough that there are a zillion implementations of it with a zillion bugs to match.

Unlike specs, which just let you find bugs earlier, tests also help you make finding (and fixing) a bug later be cheaper.

Watching a test-driven developer work can be pretty boring.  We write a test.  We watch it fail.  We make it pass.  We check it in.  Then we write another test.  After a while of watching this, a manager will get itchy and say, Jeez!  Why can't you just go faster!  Stop writing all these darn tests already!  Just write the code!  We have a deadline!

The thing that the manager hasn't noticed here is that every ten cycles or so, something different happens.  We write a test.  It succeeds.  Wait, what?  Oops!  Looks like the system didn't behave like we expected!  Or, the test is failing at a weird way, before it gets to the point where we expect it to fail.  At this point, we have just taken five minutes to write a test which has saved us four hours of debugging time.  If you accept my estimate, that's 10 tests × 5 minutes, which is almost an hour, to save 4 hours.  Of course it's not always four hours; sometimes it's a minute, sometimes it's a week.

If you're not paying attention, this was just a little blip.  The test failed twice, rather than once.  So what?  It's not like you wouldn't have caught that error eventually anyway!

Of course, nobody's perfect, so sometimes we make a mistake anyway and it slips through to production, and we need to diagnose and fix it later.  The big difference is that, if we have 100% test coverage, we already have a very good idea of where the bug isn't.  And, when we start to track it down, we have a huge library of test utilities that we can use to produce different system configurations.  A test harness gives us a way to iterate extremely rapidly to create a test that fails, rather than spinning up the whole giant system and entering a bunch of user input for every attempt at a fix.

This is the reason you don't just write giant integration tests first.  If you've got a test that just tells you "COMPILE FAILED", you don't know anything useful yet.  You don't know which component is broken, and you don't know why.  Individual unit tests with individual failures mean that you know what has gone wrong.  Individual tests also mean that you know that each component works individually before inserting it into your giant complex integrated compiler, so that if it dies you have a consistent object that you know at least performs some operations correctly, which you can inspect and almost always see in a sane internal state, even if it's not what the rest of the system expects.

Giant integration test suites can be hugely helpful on some projects, but they are the things which are sometimes unnecessary gold plating unless you have a clear specification for the entire system.  Unit tests are the bedrock of any automated testing strategy; you need to start there.

Unit tests seem like they take time, because you look at the time spent on a project and you see the time you spent writing the tests, and you think, "why don't I just take that part out?".  Then your schedule magically gets shorter on paper and everything looks rosy.

You can do that to anything.  Take your build automation out of your schedule!  Take your version-control server out of your budget!  Don't write a spec, just start coding!  The fact is, we pay for these tools in money and time because they all pay off very quickly.

For the most part, if you don't apply them consistently and completely, their benefits can quickly evaporate while leaving their costs in place.  Again, you can try this incomplete application with anything.  Automate the build, but only the compile, not the installer.  Use version control, but make uncommitted hand-crafted changes to your releases after exporting them.  Ignore your spec, and don't update it.

So put "100% test coverage" on your personal copy of the Joel Test.  You'll be glad you did.

One postscript I feel obliged to add here: like any tool, unit tests can be used well and used poorly.  Just like you can write bad, hard-to-maintain code, you can write bad, hard-to-maintain tests.  Doing it well and getting the maximum benefit for the minimum cost is a subtle art.  Of course, getting the most out of your version control system or written spec is also a balancing act, but unit tests are a bit trickier than most of these areas, and it requires skill to get good at them.  It's definitely worth acquiring that skill, but the learning is not free.  The one place that unit tests can take up more time than they save is when you need to learn some new subtlety of how to write them properly.  If your developers are even halfway decent, though, this learning period will be shorter than you think.  Training and pair-programming with advanced test driven developers can help accelerate the process, too.  So, I stand by what I said above, but there is no silver bullet.

You Got Your WindowMaker In My Peanut Butter

Electric Duncan mentioned Window Maker and Ubuntu yesterday, and it reminded me of my own callow youth.

Nowadays I'm a serious Compiz junkie, so I don't think I'll be switching back any time soon.  Personally, I wouldn't want to live without maximumize or the scale window title filter.  However, I can definitely see why one would want to: WindowMaker is lightning fast, as well as being very simple and streamlined.  When I do pair-programming that needs tools that won't run in Screen, I spin up a WindowMaker session in a VNC server.  Sharing my whole gigantic screen with all the whizzy effects is impractical over anything slower than a local 100 megabit connection.

One of the problems with switching to a different window manager for your main session these days, however, is that things unrelated to window management stop working.  Your keyboard settings no longer apply, your media no longer auto-mounts, GTK ignores your theme, your media keys stop working, and your panel disappears, along ever-so-useful applets like Deskbar and the NetworkManager applet.

But, this need not be so.  GNOME will happily accomodate an alternate window manager. All you need to do is make sure that WindowMaker and Nautilus don't fight over the desktop, and then tell GNOME to start WindowMaker.

Of course, your desktop won't be quite as lean as if you'd eschewed GNOME completely.  It's up to you to decide whether these features are worth a few extra megabytes of RAM.

First, run gconf-editor and turn off "/apps/nautilus/preferences/show_desktop".  This should make your desktop go blank.

Next, you need to go to "System → Preferences → Sessions", and hit "add" on the "Startup Programs" tab.  Add an entry for WindowMaker:

Now, all you need to do is log out!  You will, of course, want to tweak your panels a bit when you log back in, but that part's easy: right-click and season to taste.

A Meandering Review of the Logitech Illuminated Keyboard

I haven't done a keyboard review in quite some time.  Partially this is because I've started getting only higher-quality keyboards, and so I've been getting them less frequently.  I can reliably destroy a cheap-o dome-switch keyboard in about 6 months, so now I only buy keyboards with mechanical or scissor switches.  (My rule of thumb is that if it doesn't tell me how many keystrokes the switches are rated for, I won't get it myself or put it on my wish list.  Typically the lowest number you'll get is "five million", which is a good deal higher than the two million that most dome switches can do.)

This Christmas, my grandmother kindly bought me a Logitech Illuminated Keyboard, which I've been using for the past few weeks.  I have to say I'm very happy with it.

Tactile Response

First and foremost, of course, is the keyboard's feel.

I generally prefer aggressively clicky keyboards like the Das Keyboard or the venerable Unicomp EnduraPro (known in a previous life as the "Model M").  However, at home, these are not an option, as I have both limited vertical space underneath my monitor and a limited acoustic tolerance.  Some amount of "click" is a requirement though, or the lack of feedback causes my hands to tense up and hurt.  Just a few days ago, Cyril Kowalski of techreport.com described my experience almost exactly, in his review of the Das Keyboard.  I feel this is really worth repeating:

So, because dome-switch keyboards don't let you hear or feel exactly how much force you need to depress a key, you might find yourself pushing too hard or too softly. That can mean either more fatigue or more typos. Some users try to alleviate those shortcomings with split ergonomic keyboards, which place your hands in a more natural position, but those don't really solve the feedback problem — although they can feel comfy enough to type on.

I don't have any statistics handy, but I can throw some anecdotal evidence at you. (Take that however you please.) I've been typing 2,000 words a day five days a week for around three years on a 1989 Model M, and my fingers, hands, or wrists never get tired. When I was using a Microsoft Natural Keyboard Pro and typing less each day, I suffered from finger pain and annoying wrist tingling on a regular basis. I actually type faster on the Model M, as well, even though my touch-typing technique hasn't changed.

While I've varied the keyboards I've used considerably more than Mr. Kowalski apparently has, my experience typing lots of hours on a mushy Microsoft Natural Keyboard, at Origin, was exactly the same.

So, does the Illuminated Keyboard stack up?  In a word, "yes".  But you all know that I wouldn't use one word where 500 will do.

With "illuminated" right in the name, one might think that this keyboard is a gimmicky one-trick pony.  I've defnitely seen a few other keyboards where some marketing genius duct-taped a couple of 2¢ LEDs to the back of a crummy keyboard, spray-painted the word "GAMING" on the box, and marked it up by $50.  Even Logitech's own prior entry into the "illuminated" arena, the G15, suffered from this overfocus on bling.  Here, I'll have to amend my own review: while I was impressed at first that the G15 had reasonable tactile feedback, especially for a dome-switch keyboard, it degraded over time, as any dome-switch keyboard will.  The marketing copy talks about illumination and LCDs, but doesn't mention what type of key switch is used.

(While I'm trashing on the old model, I should also note that the "G15" that Logitech is selling today has been visibly upgraded in a number of ways, and may use a new key system as well.  Given that the new G19 costs $200, contains a USB 2.0 hub, a 320x240 color LCD, and a computer that runs linux (I am not kidding), I am hoping that it doesn't ship with keys that will wear out after a few months.)

The Illuminated Keyboard, by contrast, dedicates half of its marketing copy to talking about the key switches.  Like the diNovo Edge (but at less than half the price) it uses the "PerfectStroke Key System".  Indeed, the keyboard's tactile response feels like an updated version of the Edge.

I also have an Edge, and I am quite happy with it too.  If anything, the keys on the Illuminated Keyboard are even better calibrated.  While scissor-switch keyboards are all fairly similar, I have managed to beat many of my own speed records with this keyboard, and a few brief experiments side-by-side with the Edge suggest that I can type as fast or slightly faster on the Illuminated Keyboard.  The keyboard I was most recently using on this computer was the Moshi Celesta (warning!  link contains obnoxiously huge animation, and plays music).  I can type at about the same raw speed on all the scissor keyboards I've tried (the Celesta, the IceKey, the Edge and this one).  However, I have a marginally, but consistently, lower average error rate with the "PerfectStroke"-based keyboards.

This tactile similarity gives me high hopes for the durability of the Illuminated Keyboard as well.  When I first got my Edge, I hammered on it as my primary keyboard for a good eight months.  This is more than enough time to kill lesser keyboards.  Then, we moved it to the media center, where Ying and I would still both use it daily.  As far as I know, it would have lasted another five years, but some part of the battery or the charger gave up the ghost and it would not recharge.  (No complaints there, though.  Logitech replaced the whole unit, free of charge, despite the fact that it was out of warranty.)

Illumination

So, it feels pretty good.  Now, on to the headline feature.  Is it illuminated?

Yes.  The illumination is fairly subtle even on its brightest setting.  It's white, not some neon flourescent color.  It's not nearly as bright as many "gaming" keyboards.  However, it's also very even.  I'm not sure if they use the same trick that Déck does and backlighting every key individually, but there are no dim spots.

Actually, a better answer would be "only if you want it to be".  Regardless of whether you like backlighting — in fact, even you find backlighting obnoxious — this is a pretty good keyboard.  It has a button which allows you to select a light level.  You can turn off the light as soon as your computer starts up, and leave it off.

Design

The form-factor and design of the keyboard are also satisfactory.  As you can see on logitech's site, it's very thin, flat, and it has an integrated wrist-wrest.  The texture of both the keys and the wrist-wrest is slightly rubberized, which keeps my wrists comfortably in place and prevents my fingers from slipping onto adjacent keys when typing quickly.

The {caps,num,scroll}lock keys are vanishingly unobtrusive, but unlike the the Edge's ill-considered "boop-BEEP" audio replacement for the LEDs, they are present and visible.

Of course, any keyboard review would be incomplete without a consideration of "special features".  Normally I find "multimedia keys" and unusual layout options a grating misfeature.  For example, on my Moshi Celesta, there is an "Eject" button immediately underneath "Page Down", which I would accidentally hit at least once a day.  To my surprise, the Illuminated Keyboard is the first one where I've really used the "multimedia" functions.  They are unobtrusive.  The only dedicated "special" keys are far to the right, where there are volume controls and the button used to adjust the keyboard's backlight.

Most of the multimedia keys are alternative meanings for F1-F12 and PrintScreen/Pause.  Much like on the Edge, an "FN" key replaces the right-windows key.  Holding FN while pressing a function key invokes its alternate meaning.  For example, there is a "previous track" icon above F10, so if I press FN-F10, my media player skips back a track.  Despite a similar setup on the Edge, I never really used the multimedia keys there, because it's awkward to move my right hand so I can hit "FN" with my thumb, then reach over with my left hand to hit the appropriate function.  On the Illuminated Keyboard, the FN key is considerably wider, and the functions that I actually want to use (Previous Track, Play/Pause Music, Next Track) are located on the right hand side of the keyboard, which allows me to easily hit them by moving only one hand.

Of course it didn't hurt that I discovered the "multimedia keys" plugin for my music player at about the same time.

The layout is a tiny bit nonstandard, but in a very useful way.  The seldom-used "insert" key has migrated north to a less prominent position on the "function" row.  In its place, the "del" key has expanded to take up two spots.  Again, I don't like layout tweaks, as they often do more harm than good, but this prevents a common and irritating accident, hitting the "insert" key when I intended to hit "delete".  (I don't know why this never happens with "Home" and "End" or "Page Up" and "Page Down", but it is a real problem.)  Aside from that, this is a bog-standard PC 105-key layout.

Annoyances

Obviously I'm pretty happy with this keyboard, but I always find the most useful part of any review the "why not" section.  So, what's wrong with this keyboard?  With this one it's a pretty short list, but it's not empty.

1. Very occasionally, the space bar squeaks slightly.  I've had this problem on a number of different keyboards, since the wider spacebar necessarily needs a different switching mechanism, usually propped up by a small metal bar.  This isn't a huge bother.
2. The plastic of the keyboard is bowed slightly, such that the rubber foot in the middle of the keyboard doesn't quite touch my desk when it's laid flat.  This means that the keyboard warps a little bit if you rest any weight on it.  This might even be intentional (some kind of ergonomic consideration?) but the slight warp seems like a flaw in otherwise quality construction.
3. What I think are the "Instant Messaging", "Switch Window", and "Run" function buttons don't seem to register on Ubuntu.  I don't know if this is a problem with the keyboard, GNOME, Linux, or what, but I wish I didn't have to know.  (I was impressed to note that all the other keys seem to do something useful out of the box.)
4. The Alt keys are a tiny bit too narrow for my taste.  Of course, being an Emacs user I have a strong bias towards having an overlarge meta key, so YMMV.  Most people who use the "control" key in the wrong position would probably disagree, as the slightly narrower Alt keys are that way because the "Ctrl" keys are nice and wide.  That said, looking at the keyboard I thought this would bother me, but in practice it hasn't.
   

Conclusions

The most obvious conclusion we can draw here is that I think and write about keyboards way too much.  Beyond that, the Illuminated Keyboard would get very high marks, if I did numeric grading here.  I think it will replace the MacAlly IceKey for my default keyboard recommendation.  It's got all the same properties (quiet, small, low-profile) which made that keyboard a good recommendation.  However, the construction is apparently higher-quality, it has more function keys (despite the fact that they don't get in your way if you don't use them).  The illumination is a nice touch: even us touch-typists can't necessarily remember where the "eject" button is in the dark.

Thanks, Grandma!

Commercial Break

As long as I'm doing all this blogging, there's a post I've been forgetting to do for months.  I'll keep it short and sweet:

At Divmod, we do consulting, including performance analysis, custom development, and open source maintenance.  If you have problems that involve Python, Twisted, or any Divmod open source project, you're unlikely to find better.

I usually handle inquiries, but at the moment, I'm working on some secret projects of my own.  If you're in the market for one of those things I mentioned, you should get in touch with JP Calderone.

(I am periodically amazed that people close to me don't know that we do consulting.  I need to remember to get out there and toot the horn every so often!)