Open source is, at least in part, about you, where “you” refers to the user.

Open Source Is Not About “Open Source Is Not About You”

In other words: Rich Hickey was wrong when he wrote “Open Source Is Not About You” and I’m tired of pretending otherwise.

Of course he’s not completely wrong, or his famous post would not have resonated quite so much in the first place. Obnoxious users who demand their personal use-cases be immediately addressed by volunteer maintainers for free should indeed be viewed as the pariahs that they are. Similarly, corporate users who want free support from the community that supplies their infrastructure to lower their costs. As should those who profit from this type of externalization by their own customers.

But the exchange of “open source” (or even “free software”) is not as simple as “I have prepared some software for you, please enjoy it, you have no right to complain”, and maintainers ought to have a precise understanding of the costs and benefits — as well as the ethical implications — of that exchange.

Right now we barely even articulate that the exchange exists, let alone that it establishes a long-term, subtle, and implicit relationship between maintainer and user.

Let’s fix that.

What Are We Doing When We Do An Open Source?

Hickey, and his many acolytes in the years since his fateful post, asserts that the process of “open source” goes like this:

1. Maintainer makes a thing, and makes it available to users as a gift.
   1. Maintainer may “love working with the team”.
   2. Maintainer may be “proud of the work we do”.
2. Users accept the gift, and extract utility from it.
   1. (Users MUST be grateful for this.)
3. A tiny fraction of users reciprocally contribute to the thing.
   1. (Maintainers may be grateful for this.)

He makes various oblique references to the specific activities of his company, which does things vaguely related to his projects for money¹. These activities are exclusively characterized as for “customers”, however, a subset of the aforementioned users so tiny (“fewer than 1%”) as to nearly be an entirely distinct group.

Breezing past this process in an essay about obnoxious users demanding things they are not entitled to, one might nod along, as this sounds mostly sensible. Giving gifts is nice. I too love working with good teams and taking pride in things.

Examined more closely, however, it starts to logically fall apart. If you have consulting clients and that’s where all of your money is coming from, why are you bothering (as he repeatedly insists) “doing [things] for the community”? What was the point of releasing this code in the first place? You could love working with your team and be proud of the work that you do in a lot of different contexts; why bother implicating this horde of entitled and obnoxious people, if that’s all you’re getting out of it? What’s in it for you?

If we’ve left out something as fundamental as “why is the maintainer doing this”, perhaps this story leaves out some other important bits as well.

Why Are You Doing This?

There are many possible motivations for releasing and maintaining open source software. They are often subtle, often overlapping, and rarely clearly stated. Maintainers are not a monolith and not everyone does it for similar reasons. But let’s review a few reasons that someone might want to contribute.

Reputation

One reason that you might want to release some open source software is advertising. The most common form of this is self-promotion; if you are a visible, prominent contributor to an open source project, it stands to reason that you will have an easier time finding work in the domain of that project.

If you operate a consultancy, as Rich Hickey did at the time of his famous rant, then this reputational currency translates into advertising for your services. It’s a practical demonstration of the skills of your team.

The trade in this benefit is most like the traditional “gift economy” that open source has been compared to. You give the code to your users, which has some value, but the users give you back some reputation, in the form of their attention, their esteem, and possibly even their money if they become customers or employers.

Influence

Infrastructure is the most popular type of open source for a good reason. Programmers working on a problem are often hemmed in by sclerotic architectural choices which prevent them from solving problems in the way that they’d prefer to solve them. Major infrastructural investments are difficult to justify in a planning process, as their benefits are hard to prove. Sometimes the benefits are highly personal; different engineers have different aesthetic preferences about what types of equally-valid solutions they’d prefer to work with.

If you can develop your preferred type of solution and release it as open source, then you can influence how everyone else solves this type of problem. As an individual, such a position of influence can allow you to have some transferable expertise between employers. You know how to use the tool you developed, so you can be very quick and effective with it, and you can shape it to your ongoing taste over time.

If you’re an employer, and you can get everyone else to use your open source thing², this can reduce both your hiring and training costs. Potential employees can read the code, see that it’s good, and want to work at a place that produces good code like that. They can also read the code and become familiar with it in advance of coming to work for you, which means that you have a ready supply of developers who already know how your internal systems work.

The trade in this benefit is more like “soft power” than a gift economy. You give the code to your users, which has some value, but the users give you back the ability to dictate their technological agenda. You gain both the ability to influence their initial direction, and, as part of ongoing maintenance, to dictate their behavior over time.

Improvement

As an engineer, you might want to improve your own skills. Writing something proprietary and commercial cuts against this in two ways.

First, you will want to build something that already exists within your skill set, so that it will attract commercial interest and actually be competitive. Within the context of a larger team, you will want to personally be able to be immediately effective for similar reasons. But you still need a way to learn new things.

Second, you will want to build something somewhat secretively, so that the value you are producing is captured rather than released to the community. This means that you will be cut off from external sources of expert feedback.

As an organization, you might want to build the skills of your staff in similar ways.

The trade in this benefit is code for knowledge. You release the code or changes, and in return you expect your users to provide you good bug reports, and to induce at least some of them to become co-developers.

Outsourcing

As an engineer, you can only do so much on your own. Perhaps you want to have some influence over your infrastructure so you want to write it, but you also want to have a communal place to keep your infrastructure such that you can make a change to something to suit your needs, but you know that even if you walk away, someone else will maintain that change and keep it working across years or even decades of changes to underlying platforms, hardware, etc.

This sort of communal maintenance effort can be shared among all interested participants; if a thousand companies all need the same tool, if even a few dozen can share it, that reduces even their own load massively, let alone everyone else’s.

The trade in this benefit is more complex, since there’s less symmetry between the main maintainer and peripheral community members who also contribute code. The main maintainer is actually trading a namespace, a central place for people to contribute, coordinate, and release changes, rather than the code. They are a sort of market maker where then all the other contributors trade code for code within that market-ish structure.

In practice, this motivation produces a game theory problem where, when maintenance drops below a critical threshold, it creates a big enough crisis that at least some freeloading stakeholders will be forced to start making contributions.

Ultimately, however, this saves all involved parties a ton on maintenance, more eager volunteers who do not freeload in the first place get all the other benefits mentioned above as well.

The Things that Aren’t Supposed To Be Benefits

These categories could be made as explicit, rational trade-offs, even if they are often implicit and subtle in practice. They are transactions where the maintainer gets something and the user gets something.

However, not everything that you are getting as a maintainer is something you are actually supposed to use to your own benefit. Being given trust in service of a responsibility is not a transaction.

“Oops, All Root Shells”

Open source code is code. In our modern world of absolutely pathetic sandboxing, installing code from somebody else gives them control over your system, even if it is somewhat indirect.

There is an unwritten rule that if I create an open source library, and you use it, it probably shouldn’t have a backdoor in it that gives me the credentials to your bank account. There is a trust relationship between the user and the maintainer, and here, we see the first obligation that the maintainer has. The maintainer is obligated not to use the user’s computer for their own gain.

This rule might seem obvious and straightforward. It might even seem unfair to you that I call the rule “unwritten”, because the rule is, in fact, written down in a few places: for example, in the npm Acceptable Content Policy, it says right there:

A few examples of unacceptable content:

…

3. Content containing malicious computer code, such as computer viruses, computer worms, rootkits, back doors, or spyware. This includes content submitted for research purposes. Tools designed and documented explicitly to assist in security research are acceptable, but exploits and malware that use the npm registry as a deployment or delivery vector are not.

I think we can all agree that a script which steals your bank credentials and sends them to me to buy a totally sick jet ski would qualify as “malware”, so clearly that is forbidden.

There is also an enormous gray area here. npm also explicitly allows “Information on how to pay, donate to, and otherwise support Package development”, but then goes on to explicitly forbid “Packages that display ads at runtime, on installation, or at other stages of the software development lifecycle, such as via npm scripts.”⁴ How are the lines drawn around these gray areas? “npm will continue to apply its judgment when deciding what content is acceptable.”

But also... this is forbidden by npm, not by the transcendental nature of “open source”. I could give away code that displays all kinds of ads to its users as a “gift” on my website. The exact structure of this policy is not uncommon, but it also isn’t exactly the same as other such sites. PyPI, for example, explicitly bans “cryptocurrency mining”, which NPM does not. Is cryptocurrency mining “not open source”? A lot of judgement calls are happening here about what is allowable in these “gifts” that you are giving to your users.

But I digress.

My point is that policy-making around this concept is not clear, there are lots of little disagreements around the edges, but there is a very strong consensus that while the user is giving you their trust here, that is not a trade. The deal is not “you give the user some code, the user gives you unlimited compute and access to all their financial accounts”. The user has made themselves vulnerable to your code on the strength of your reputation.

This creates an obligation for you to not do anything evil with that code, either intentionally or through negligence.

Security Updates Are Just Command And Control In A Funny Hat

All of this is just about the initial download of some code, and that is the way that Rich Hickey describes it, as if you just grabbed some code off a web page and put it in a folder that you like on your desktop. But that is not how open source relationships work today, if indeed it ever was.

The way it works today is that you add a dependency to your pyproject.toml or your package.json or your Cargo.toml and now your users are vulnerable not just to whatever you happened to upload in the first place, but to whoever happens to have your package index credentials.

This creates an obligation to maintain an operational security posture that protects your users from malicious updates.

The Roadmap Is Someone’s Life

Another kind of trust that the user is placing in you is the trust that you are going to have at least some kind of regard for their usage of your software.

In a perfect world, the user’s expectations could be clearly circumscribed. Whatever ongoing maintenance you commit to perform would be encapsulated in clear policies that you’d write up in advance, about exactly what kind of security response policy you have, how you will communicate when you no longer have the resources for maintenance, and so on.

But anyone who has been involved in any project at anything but the most extreme tier of operational maturity knows that 99% of the ecosystem relies on a set of loose conventions around how all that stuff works. We expect that maintainers will generally be around, that they’ll use existing tools like an issue tracker for triaging user bugs, GHSA and CVEs for security reporting, that they will mark the project as “archived” and maybe do a final release before abandoning it, that they will maintain a ChangeLog explaining at least a little bit of what is going on.

Users assume that those conventions will be followed when there are any gaps in explicit policy, or indeed if policy is lacking entirely. This assumption is reasonable, because otherwise nobody could ever use any open source without a stack of service contracts that nobody has any time to write.

The strongest such convention is that an actively maintained program will, at least, more or less keep doing what it does as time goes on. A user who has elected to use a bit of open source software has made themselves vulnerable to changes and breakages in that software by the mere fact of using it. In the time that they have used it and invested in it, they have not invested in:

• creating alternative software to meet their needs,
• maintaining data in formats that other software can read, or
• learning how to use existing alternative software.

This can, and does, go badly wrong, when those expectations are mismatched.

How It Goes Wrong

Let’s say a maintainer creates an open source paint program, OpenPaint.

An artist, known for their unique style of making blended collages, switches from their previous app, ProprietaryPaint, to this new OpenPaint to make these culturally significant works of art. However, the maintainer decides that the ‘blend’ tool is kind of a pain to maintain, and they remove it in OpenPaint 2.

A few months later, the artist’s operating system vendor issues a security update that breaks OpenPaint, because older versions of OpenPaint were unknowingly abusing some platform API.

The maintainer releases a new OpenPaint 2.0.1 that addresses this incompatibility, but doesn’t care about version 1.x any more so they don’t bother to update that one.

This places the artist in an impossible situation. They can stay on an old version of their operating system, putting all their personal data at risk. Or they can upgrade to the new operating system, effectively either cutting off access to their livelihood, or forcing them to change their art style entirely.

Now, proprietary software can place users in similarly untenable positions (and in fact, it is more often proprietary software that does). But does the openness completely remove any obligation for this consideration? Should the OpenPaint team have to at least communicate the reasons for doing this, to give the artist some recourse?⁵

The only thing that “open source” does is that it allows the artist to pay a prohibitive amount of money to a new maintenance team to create a fork. This is rarely the kind of thing that individuals can manage.

This creates an obligation to at least consider how your users might be relying on you.

This is the most complex obligation of the bunch. Obviously it does not entitle every single user to infinite work from the maintainer, but it also shouldn’t entitle the user to nothing for having trusted these subtle implied claims that the maintainer is making by making their work public.

It is a nuanced and ongoing negotiation and I do not think we have a clear moral intuition about how it should work out. But we do need to figure out a way to work it out.

It also raises a clarifying question.

Why Are We Even Doing This, and Who Are We Doing It For?

People generally like to do things for more than one reason. We live in an economy where people need to make money, but we mostly prefer to make that money doing things that are useful, and that make other people happy.

So, yes, we create open source for self-interested reasons to improve our reputations, to improve our skills, to increase our influence and to share our maintenance burdens. In so doing we take on some level of obligation to not abuse the trust that is placed in us, even if that level of obligation is not clear.

But if we are not doing it to serve those users at least a little bit, then those motivations are going to quickly ring hollow. We will not increase our reputation with a person if we respond to their every request by telling them that we owe them nothing and that their opinions are worthless. We will not gain influence over a community if we ignore their desires.

Many interactions with open source maintainers are unnecessarily adversarial. This is of course partially the fault of those users, who should calibrate their expectations appropriately.

Still: maintainers could do a better job of listening before these interactions become toxic. There’s no reason that “open source users” should be an especially toxic group of people. At this point in history, that group is basically just … people with computers.

It’s like that old truism. If you meet one person who is a jerk to you, that’s their problem. But if everyone you meet, everywhere you go, is constantly abrasive to you and treats you like you’re doing something wrong, maybe it’s time to look inward.

If all open source users are entitled assholes, maybe it’s time to look for a structural problem.

Surprise, It’s About AI Again

Sigh.⁶

Users hate slop.

I know, dear AI-positive reader, your AI outputs are different from everyone else’s, you aren’t pushing thoughtless slop into your code, just because everyone else is and it is the inevitable terminus of using those tools. You aren’t “lazy vibe coding” with Claude, you’re doing “responsible agentic engineering”, which is different because you’re just built different.

Still, humor me, for a moment. Your users don’t know that. They know what it looks like when products that they like adopt slop. They know that they will start leaking data. Developers know that it will make them personally less secure. They know that they can expect more outages and that your code will inexorably decline in quality.

In other words, your users are going to assume that this means you are violating that final obligation that the software should keep working.

Your users are going to tell you to stop, and they are probably going to get mad. Maybe you, or a plurality of your team, also want to stop, maybe you disagree with them, but in any case you need some way to have that conversation in a way that does not immediately overflow into every adjacent discussion forum. Users need to feel welcome in some space so they can have the discussion in that space, and not explode out into a thousand different group chats and social media threads.

This post was inspired by yet another prominent open source community discourse where a ton of angry users showed up to yell at developers to stop accepting LLM-generated code. I’m not going to link to any of these, because we don’t need any more fuel for the discourse fire. But there is more than one such case and the pattern is becoming familiar.

On social media - usually BlueSky or Mastodon, but sometimes a user group forum - users become aware of some AI-adjacent policy. They show up in a horde to the developer forum or mailing list. They loudly start demanding the project take a hard stand⁷ against AI. This pressure is simultaneous, but uncoordinated; extremely repetitive, very diverse, often inconsistent, and pretty stressful, especially if you’re a burnt-out maintainer with other things to be doing who may not even like AI yourself in the first place.

Believe me, I get it. It can be very unpleasant to deal with.

Like most problems that AI is causing, though, it’s not really an “AI” problem as much as it is a pre-existing dumpster fire that “AI” is pouring gasoline onto. In this case, an online mob is the language of the unheard⁸.

If Users Are Mad It’s Probably Already Too Late (But Maybe You Can Get Ready For Next Time)

One day, all of a sudden, you’re getting feedback from a bunch of users that are using inappropriate channels to complain. But did they already have appropriate channels to use?

Did you have a place for people to congregate and discuss your project? To make orderly complaints in a way that will be legible to you? Or do you just have a GitHub Issues page, which non-technical users have no idea how to interact with, and a forum for developers, where users don’t know the norms and any arriving brigade of pissed-off users will be seen as disruptive and inappropriate?

I don’t want to be throwing any stones from within my particular glass house. Setting up such a place has gotten harder over the years. I don’t really have one, either.

Could I have one, though? IRC has been slowly dying, mailing lists are unpopular and present increasingly annoying moderation challenges, forum software is expensive to operate and keep maintained, Discord is a confusing mess and the upshot of all of this is every community needs community management and forum moderation. Which means that for my own small solo projects, I couldn’t possibly have such infrastructure because such infrastructure requires a dedicated second person to maintain it, and until someone volunteers for that, it’s not really feasible. Even for my larger projects you’d be surprised how slim of a skeleton crew we are getting by with, and we definitely don’t have a whole spare maintainer to go manage this, especially as we are under attack from the slopocalypse ourselves.

The nature of open source community is that most communities start too small to need such a thing, grow incrementally until one day they are suddenly way too big and needed one yesterday, and then suddenly they are too small again when interest wanes even a little bit. Even as we need it more and more, building and maintaining community infrastructure remains a challenge.

Even so, having a dedicated place for users — not maintainers — to converse amongst themselves, be an actual community, and present feedback to the developers, is fast becoming a necessary component of a successful community and not a nice-to-have.

In Conclusion

As trying as it can be sometimes, we maintainers all do get something out of open source, and it is good to be honest with your users — and with yourself — exactly what you want to get out of it. In order to know whether the juice is worth the squeeze, we must know both what the juice is, and what the squeeze is.

Part of the metaphorical squeeze is a set of obligations, and those are the most poorly defined of all. We should try to be clear about what those are too. Both about exactly what we believe we are signing up for, and also, about how we are willing to let our users hold us to account for them. Codes of conduct are a start here, but only the absolute barest bare minimum; “do not harass your colleagues or your users” is not a standard of excellence to aspire to, it’s just basic manners.

I can’t tell you exactly what your obligations are, only try to gesture at my idea of the outlines of the fuzzy moral intuition we’ve all been implicitly sharing up until now.

Drawing this line is not just for the benefit of the users, either. Maintainers already feel pressure, we already feel obligations. We resent that feeling of obligation. While there are a diverse array of reasons for that resentment, one big one is that it’s not clear, even to ourselves where the obligations end. Lashing out by saying “I promised nothing and I owe you nothing!” followed by some choice expletives feels cathartic, but it doesn’t really solve the problem, because we clearly don’t really believe that’s where the line is, or we would have already stopped there. We wouldn’t feel the need to say it.

It is going to be a very big collective endeavor to figure out exactly where that line is. The best time to have gotten started on that endeavor was 50 years ago.

But the second best time is today.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor!⁹

I get asked this rhetorical question a lot, in various forms:

Sure, datacenters might use a lot of energy, but you don’t have to use a hosted frontier model to do software development. What if I just run a local open-weights model to do some coding, with an open-source coding agent? Video games also use my GPU. Is local model development any worse than playing a video game?

So I want to write down my comprehensive answer to this: Yes, using an LLM to write some code is worse than playing a video game, for a few reasons.

Video Games Are Interactive, LLMs Are Batch Jobs

Video games use compute to respond to human input. You are using your GPU while you are looking at a screen, displaying an image. When you are done playing, you shut off the game, and your computer goes back to idle. It’s much less energy. By contrast, agentic loops with evals (the only kind of “AI” that is meaningfully any good at coding) are running hot, for days. To use the most recent example of such a thing, a very rough first sketch of an implementation of a Windows graphics API backend to help port a paint program to other platforms, it took 3 weeks of Claude time, “day and night”. Do you play a lot of video games for 500 hours to make it past the tutorial level, while also using other computers for other things, as well as the rest of your carbon footprint?

Video Games Need Development, LLMs Need Training

Video games use compute to respond to human input during development, too. Your game has to be made, but your LLM has to be trained. LLMs use a historically extreme amount of power, probably using more than the entire Internet, but it’s kind of hard to say. Still, it seems a reasonable estimate to within several orders of magnitude that even over a multi-year project with hundreds of developers, the power used to develop an individual video game is nowhere close to training even a small LLM.

This is true even for local models. OpenAI has openly claimed that DeepSeek “stole its intellectual property”, and I have heard grumblings that none of the open-weights generalist models could realistically exist without the massive lift that the frontier labs are doing with their training, in various other ways too. Secrecy throughout the industry makes this kind of impossible to understand rigorously, but it seems fair to say that you are partially culpable for all that famously energy-intensive frontier lab training if you’re using a local model.

And They Keep Needing Training

You also can’t dismiss this as a sunk cost, because in order to stay current with industry developments, models need to be updated with new information from the rest of the world, which means that you need to keep training them. Beyond the energy for your own use, if you want a real-life agentic workflow that actually does useful stuff, practically speaking you would still need to update your local models over and over again, at least once every few months, which means you would be incentivizing continued energy consumption by whoever was doing that training for you, including the energy cost of scraping.

Let’s Be Real Here, You Aren’t Actually Using A Local Model

This question is a hypothetical thought experiment. Despite synthetic benchmarks that keep showing there isn’t much difference between open weight and frontier models, nobody’s actually using local models for much of anything beyond sharing those talking points. Depending on which benchmark you’re looking at, maybe it’s good enough or maybe it’s worse.

As an inveterate AI hater, all these systems seem pretty bad to me, but it seems that people who find them useful tend to subjectively believe the frontier models are worth the premium, and that’s what they’re actually using. Once you have accepted that it is OK to use LLMs for coding at all, it seems like a very quick slippery slope on down to “we’ll go ahead and use the frontier models for now anyway, but we could be ethically better in the future by switching to an open weights one, that option is always available”.

There’s A Reason We Have Data Centers

Devolving power usage to local LLMs might be good to make users responsible for their costs and decrease the impacts to communities that are physically next to huge concentrations of power utilization, not to mention generation. However, there’s a reason that it makes sense for the providers to build these giant facilities: economies of scale reduce total power consumption, they don’t increase it. If you do all the same stuff with a local model that they have to do in hosted environments, it will probably take more power, even though you will be incentivized to do different stuff. This incentive to “do different stuff” is why although local models can hypothetically hold their own against the frontier labs for some tasks, when people or businesses take their inference costs in-house they often find that it’s too painful and move back to hosted LLMs.

There Are Problems Other Than Power

These are subjects for a different post, but you have to consider a lot of other externalities: AI psychosis, de-skilling, comprehension debt, cultivating a dependency, introducing security defects, limiting your design space based on what LLMs can understand, context rot, wasting time on invalid solutions, introducing unpredictability into your workflows. You still have to consider the total cost benefit ratio.

To Sum Up

Local LLMs might alleviate some of the harms from using the hosted frontier providers. There are fewer privacy concerns, you can measure your power utilization and be more directly responsible for it, you can build interfaces with affordances that are less oriented towards addiction and dependency than the major frontier labs’ harnesses.

But they’re not automatically “the same as playing a video game” just because they can use the same GPU.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor!

As I have discussed in previous posts, “AIs” can make mistakes. In fact, they do make mistakes, and their mistake-making patterns are such that where and how they will make mistakes is both uncertain and constantly changing.

Thus, in any scenario where you want to attempt to make “productive” use of “AI”, you must have a system in place for checking every result. Not checking some results; checking every result. If each result might have a consequence for you (and if it didn’t have a consequence, why bother automating it?) and you cannot predict in advance which kinds of results will need verification, then verification is always required.

The verification often ends up being just as expensive as doing the work in the first place, which means that if you want your usage of “AI” to be personally profitable, you have to find someone else to externalize the cost of verification onto. This person becomes your adversary, and, if you are successful, your “AI’s” victim.

The Ladder-Climber And Their Reverse-Centaur Rungs

One way that this constellation of facts can straightforwardly assemble themselves into a dystopian nightmare is the phenomenon, described by Cory Doctorow, of the reverse centaur. This is when your employer non-consensually turns you into the verification system. The “AI” does the fun part of initially performing the work, and then you do the boring part where you check if the robot is right and clean up its messes, even if everyone already knows that it would, in aggregate, be cheaper for you to do the work in the first place.

Reverse centaurs can be made from any automation, not only “AI” automation. I think that there is a reason that this term happens to have emerged in the “age of AI”, though, and not with earlier automation technologies (even those which were considerably more viscerally horrific). That reason is: the wrongness of “AI” output is not merely a technical feature that must be compensated for, it is a generalized externality.

As I mentioned above, if you are responsible for the entirety of the work, both extruding the “AI” output and checking it, it’s usually cheaper to have humans do the entirety of the work to begin with. When humans do the writing directly, we can check as we go, and thus verification doesn’t need to be as comprehensive.

When “AI” coding advocates say “code review is the bottleneck”, what they are observing is that the LLM is still rolling the dice for each PR, and a human is still necessary to verify that each of those rolls is a winner. But calling this process “code review” is a bit of a misnomer; it’s not really “code review” in the traditional sense, it’s human understanding.

Before the advent of “AI”, the human understanding was implicit in the process of writing the code in the first place¹, and the code review was a way of diffusing and extending that understanding. Now that the code can be authored with no initial understanding taking place, that cost has not gone away, it has moved.

Human understanding was always the bottleneck.

However, this is taking a collaborative view of a software project, where satisfying the needs and solving the problems of your customers are the goals. We can see that “AI” is a bad tool to satisfy those goals, because all it’s doing is converting the first half of the work, that of understanding the code as you write it, to understanding the agent’s output as you read it.

What if, instead, we were to take the view that every software company is a Hobbesian nightmare, red in tooth and claw? In this view, the only goal of a software project is for the individual developers to make their promo cycles and get their bonuses. Given that there is only a certain amount of money to go around, this is a zero-sum game where each programmer wants to look more productive than their colleagues.

Pretty much every organization finds it easy to reward “productivity” as expressed by lines of code emitted, but the benefits of doing thorough and thoughtful design, analysis, and code review very difficult to reward. In this world, an LLM is an invaluable tool for the sociopathic ladder-climber, particularly if your legacy organization is still structuring their workflows as if the person prompting the bot is “writing” the code, and then they get to foist off the act of “reviewing” the code onto someone else.

Here, the prompter effectively externalizes the cost of the LLM’s failures but internalizes any benefits. The prompter will vibe-code a big feature, so large that the assigned reviewer can’t possibly comprehend it all effectively. When this happens, the reviewer will, eventually, be pressured to approve it, even if they can try to spot a few problems along the way. The reviewer has their own work to get back to, after all, the obligation to review the prompter’s (read: the bot’s) code is a drain on their time that they are not going to get rewarded for.

If this feature is a big success, the prompter gets a promotion. If it causes a big issue, well, the reviewer must not have been careful enough.

This is why LLMs are “good for coding”, and also why their biggest promoters keep having outages.

The Generative Gish Galloper

Coding is the biggest “success story” of this type of adversarial communication, but it is by far not the only instance of such a thing. LLMs create a new form of leverage that can turn Brandolini’s law from a linear advantage into an exponential one. If you are engaged in a political debate where you want to overwhelm the other side in nonsense, an LLM can generate bullshit faster than it is physically possible for a human being to type, let alone respond thoughtfully. There is an asymmetry to the utility of this weapon as well: only one side of the political spectrum wants to flood the zone and destroy trust in institutions and the concept of truth. There’s a good reason that the fascists love it.

Straightforward Spam and Fraud

This is kind of obvious, but LLMs can generate lightly-customized, plausible-looking text much more quickly than any human being. This facilitates their use in fraud, spam, and scams. In a spamming or fraudulent interaction, once again, the costs are externalized onto the victim: the recipient of a spam message has to do all the work of “checking” the LLM’s output. Spammers already expect very low hit rates from boilerplate, and if the LLM can increase those percentages from 1% to 5% the technology will pay for itself; they don’t need anything like reliable accuracy.

Customer “Support”

If you have any kind of commercial relationship with a company, I probably don’t even need to mention this: customer “support” bots are a misery. Everybody knows it at this point. But customer support is usually conceptualized by businesses as an adversarial interaction, because it is a cost center. They maintain internal metrics on time-to-resolution and try to optimize them. Implicitly, this creates a dynamic where the goal of the customer service agent’s job is not to solve your problem, but to emit noise that will cause you to think your problem is resolved, or to give up, as fast as possible. Unsurprisingly, LLMs can emit this noise faster than humans can, getting those customers off the phone. But those customers will remember those interactions, and the story outside the TTR metrics is horrible.

Similarly to the situation in software development, LLMs can look very good on paper for customer support, but mostly what they are doing is illuminating the problems with the industry’s existing metrics, by turning “winning the metrics battle against the customer” into a more obvious and immediate defeat for the company’s long term reputation.

“Education”

In 2026 it is sadly a fact of life that students cheat all the time using “AI”, and that this cheating is very successful, in that the teachers find it very hard to detect.

LLMs are great for cheating on schoolwork because the student is externalizing the work of the checking onto the teachers, who are often starting at a disadvantage to begin with, at least in the US.

My view is that this is happening because of a divergence in the way that students vs. teachers (or, more accurately, “the broader educational system”) view grading.

When a student is asked to write an essay, the teachers see the effort as both intrinsically worthwhile for the student, as well as useful as a pedagogical tool to evaluate and react to the student’s progress. The student, by contrast, sees a stumbling block designed to knock them off the path to success and into a permanent underclass. It is no wonder that the student sees “AI” as useful to their own goals and has no compunction about deploying it.

There is a bitter irony that the ability to understand the inherent value of actually writing the essay on their own is the sort of thing that students can really only learn by writing a bunch of essays. There’s no way that I can think of which makes the benefit legible as long as a shortcut is available.

The net effect here is a downward spiral, where the already-wobbling educational system is sustaining an attack that it doesn’t have the resources to recover from. The individual students’ attacks against their teachers and their schools’ grading systems might appear to momentarily succeed, but they will win the battle and lose the war.

Spamming “For Good”?

Usually when we talk about someone unilaterally choosing to enter into an adversarial relationship, that’s an “attack” and for good reasons we have a negative impression of the attacker. However, I would be remiss if I did not point out that there are some cases where the relationship was already adversarial; just because you’re the attacker doesn’t mean that you are evil.

For example we might imagine use-cases like automatically filing appeals for prior authorizations against health insurance. It’s relatively well-known at this point that the main way for-profit insurers maintain their margins is by denying claims right up to the line of the policies themselves being fraud, so using a spamming tool to fight them might be entirely justifiable² in that case.

Similarly, using an LLM could be justified in a fight against a company refusing to honor a warranty. One could imagine using an LLM to immediately generate replies and escalations.

However, even in imagined cases like these, the underlying problem is that the insurers and the vendors already have a tremendous amount of structural power, so it is more likely that they will have the advantage in deploying a communications weapon like an LLM, as well as enacting policies to simply ignore any LLM-based communication that you might submit. Worse, if these strategies were to become widespread, they might provide an excuse to reject any communications by feeding them into an unreliable “LLM detector” and issuing an automated “computer says no” even to hand-written correspondence.

It is also worth stressing that these cases are imagined, as compared to the very real coworker-abuse, spam, scam, fraud, and disinformation campaigns being waged in real life today.

Therefore, while legitimate uses might exist, it’s hard to imagine that there’s anywhere they would be genuinely valuable and sustainable. In the best case “AI” will provide a temporary advantage for underdogs that will provoke an arms race which the resource-advantaged adversaries will win in the long run, in the worst case the arms race itself will cement permanent structural change that will make things worse.

“Search” By Stealing

Most of the adversarial utility of “AI” is on the “write” side, since write-amplification is more obviously aggressive than reading. But the “read” side of LLMs — summarization and question-answering — can be a form of attack as well.

To begin with, the act of reading itself is currently enormously destructive, but that’s arguably not a fundamental aspect of this technology. They could set reasonable rate-limits and respect things like robots.txt, as search engines have for decades now. They could also refrain from committing criminal levels of copyright infringement. But, today, using “AI” tools does suborn this sort of out-of-control crawling.

More insidiously, consider the scenario described in this YouTube video. The LTT Bros decided to try Linux again, and in the course of so doing, they had problems. When trying to solve these problems, they were faced with a choice: they could consult Reddit, or they could ask an LLM. Asking an LLM would “gaslight the heck out of” them, but they still found it preferable, because they would at least get an answer without getting yelled at.

Initially this sounds great. But it also means that you want to extract knowledge from a community, while mechanically eliding any values or norms that the community may want to impart as part of offering that knowledge. As someone who spent many years in a community tech support role, this is worrying. Many requests for support are people asking how to do things that will momentarily solve a superficial problem but create a long-term reliability problem or even an immediate security risk, that the question-asker doesn’t want to hear about. Consider the question “I’m tired of entering my password so much, how do I make it so my laptop unlocks automatically”. An obsequious chatbot will helpfully tell you how to do this without pushback.

But, this is also a sort of ethically murky area. The Linux community is somewhat famously, for many years now, a toxic cesspool of general hostility, misogyny, etc. It is certainly a good thing that people can get access to this knowledge without subjecting themselves to abuse. But it also means that the people with the power and the privilege to change the community for the better can just quietly withdraw, rather than fixing the problems. It also means that the positive elements of culture cannot be transmitted, and people will have no opportunity to learn about unknown unknowns.

In this case, the “adversarial” communication is with society. The thing that using an LLM for search lets you do is withdraw from society and avoid forming any personal connections. There are some personal connections which are painful and annoying, and so that can feel like a momentary balm. But the need to make connections in general is, like, the concept of society itself.

Who Am I Hurting?

LLMs are good at adversarial communication. They are so good at it, relative to their other benefits, that they will tend to make communications adversarial if you are not remaining vigilant about the possibility that it might do so. My request to you, dear reader, if you are going to use such tools, is to always ask yourself, “who might I be hurting, if I use an LLM for this?”

If you’re using an “AI”, who is its adversary? If you haven’t given it one yet, who might the “AI” turn into an adversary? Who might you overwhelm with an asymmetric amount of output, or, if you’re receiving information and not sending it, who are you taking that information from without consulting?

Figure out the answers to these questions and conduct yourself accordingly; the answer might be “yourself”.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor!

Let’s say you’re writing a Python library.

In this library, you have some collection of state that represents “options” or “configuration” for a bunch of operations. Such a set of options is a bundle of potentially ever-increasing complexity. Thus, you will want it to have an extremely minimal compatibility surface, with a very carefully chosen public interface, that is either small, or perhaps nothing at all. Such an object conveys state and might have some private behavior, but all you want consumers to be able to do is build it in very constrained, specific ways, and then pass it along as a parameter to your own APIs.

By way of example, imagine that you’re wrapping a library that handles shipping physical packages.

There are a zillion ways to do it ship a package. There are different carriers who can ship it for you. There’s air freight, and ground freight, and sea freight. There’s overnight shipping. There’s the option to require a signature. There’s package tracking and certified mail. Suffice it to say, lots of stuff.

If you are starting out to implement such a library, you might need an object called something like ShippingOptions that encapsulates some of this. At the core of your library you might have a function like this:

1
2
3
4
5

async def shipPackage(
        how: ShippingOptions,
        where: Address,
    ) -> ShippingStatus:
    ...

If you are starting out implementing such a library, you know that you’re going to get the initial implementation of ShippingOptions wrong; or, at the very least, if not “wrong”, then “incomplete”. You should not want to commit to an expansive public API with a ton of different attributes until you really understand the problem domain pretty well.

Yet, ShippingOptions is absolutely vital to the rest of your library. You’ll need to construct it and pass it to various methods like estimateShippingCost and shipPackage. So you’re not going to want a ton of complexity and churn as you evolve it to be more complex.

Worse yet, this object has to hold a ton of state. It’s got attributes, maybe even quite complex internal attributes that relate to different shipping services.

Right now, today, you need to add something so you can have “no rush”, “standard” and “expedited” options. You can’t just put off implementing that indefinitely until you can come up with the perfect shape. What to do?

The tool you want here is the opaque data type design pattern. C is lousy with such things (FILE, pthread_*_t, fd_set, etc). A typedef in a header file can easily achieve this.

But in Python, if you expose a dataclass — or any class, really — even if you keep all your fields private, the constructor is still, inherently, public. You can make it raise an exception or something, but your type checker still won’t help your users; it’ll still look like it’s a normal class.

Luckily, Python typing provides a tool for this: typing.NewType.

Let’s review our requirements:

1. We need a type that our client code can use in its type annotations; it needs to be public.
2. They need to be able to consruct it somehow, even if they shouldn’t be able to see its attributes or its internal constructor arguments.
3. To express high-level things (like “ship fast”) that should stay supported as we add more nuanced and complex configurations in the future (like “ship with the fastest possible option provided by the lowest-cost carrier that supports signature verification”).

In order to solve these problems respectively, we will use:

1. a public NewType, which gives us our public name...
2. which wraps a private class with entirely private attributes, to give us an actual data structure, while not exposing the constructor,
3. a set of public constructor functions, which returns our NewType.

When we put that all together, it looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

from dataclasses import dataclass
from typing import Literal, NewType

@dataclass
class _RealShipOpts:
    _speed: Literal["fast", "normal", "slow"]

ShippingOptions = NewType("ShippingOptions", _RealShipOpts)

def shipFast() -> ShippingOptions:
    return ShippingOptions(_RealShipOpts("fast"))

def shipNormal() -> ShippingOptions:
    return ShippingOptions(_RealShipOpts("normal"))

def shipSlow() -> ShippingOptions:
    return ShippingOptions(_RealShipOpts("slow"))

As a snapshot in time, this is not all that interesting; we could have just exposed _RealShipOpts as a public class and saved ourselves some time. The fact that this exposes a constructor that takes a string is not a big deal for the present moment. For an initial quick and dirty implementation, we can just do checks like if options._speed == "fast" in our shipping and estimation code.

However, the main thing we are doing here is preserving our flexibility to evolve the related APIs into the future, so let’s see how we might do that. For example, let’s allow the shipping options to contain a concrete and specific carrier and freight method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

from dataclasses import dataclass
from enum import Enum, auto
from typing import NewType

class Carrier(Enum):
    FedEx = auto()
    USPS = auto()
    DHL = auto()
    UPS = auto()

class Conveyance(Enum):
    air = auto()
    truck = auto()
    train = auto()

@dataclass
class _RealShipOpts:
    _carrier: Carrier
    _freight: Conveyance

ShippingOptions = NewType("ShippingOptions", _RealShipOpts)

def shipFast() -> ShippingOptions:
    return ShippingOptions(_RealShipOpts(Carrier.FedEx, Conveyance.air))

def shipNormal() -> ShippingOptions:
    return ShippingOptions(_RealShipOpts(Carrier.UPS, Conveyance.truck))

def shipSlow() -> ShippingOptions:
    return ShippingOptions(_RealShipOpts(Carrier.USPS, Conveyance.train))

def shippingDetailed(
    carrier: Carrier, conveyance: Conveyance
) -> ShippingOptions:
    return ShippingOptions(_RealShipOpts(carrier, conveyance))

As a NewType, our public ShippingOptions type doesn’t have a constructor. Since _RealShipOpts is private, and all its attributes are private, we can completely remove the old versions.

Anything within our shipping library can still access the private variables on ShippingOptions; as a NewType, it’s the same type as its base at runtime, so it presents minimal¹ overhead.

Clients outside our shipping library can still call all of our public constructors: shipFast, shipNormal, and shipSlow all still work with the same (as far as calling code knows) signature and behavior.

If you need to build and convey some state within your public API, while avoiding breakages associated with compatibility churn, hopefully this technique can help you do that!

Acknowledgments

Thanks for reading, and thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor.

Humans Are Bad At Perceiving

Humans are not particularly good at catching bugs. For one thing, we get tired easily. There is some science on this, indicating that humans can’t even maintain enough concentration to review more than about 400 lines of code at a time..

We have existing terms of art, in various fields, for the ways in which the human perceptual system fails to register stimuli. Perception fails when humans are distracted, tired, overloaded, or merely improperly engaged.

Each of these has implications for the fundamental limitations of code review as an engineering practice:

• Inattentional Blindness: you won’t be able to reliably find bugs that you’re not looking for.

• Repetition Blindness: you won’t be able to reliably find bugs that you are looking for, if they keep occurring.

• Vigilance Fatigue: you won’t be able to reliably find either kind of bugs, if you have to keep being alert to the presence of bugs all the time.

• and, of course, the distinct but related Alert Fatigue: you won’t even be able to reliably evaluate reports of possible bugs, if there are too many false positives.

Never Send A Human To Do A Machine’s Job

When you need to catch a category of error in your code reliably, you will need a deterministic tool to evaluate — and, thanks to our old friend “alert fatigue” above — ideally, to also remedy that type of error. These tools will relieve the need for a human to make the same repetitive checks over and over. None of them are perfect, but:

• to catch logical errors, use automated tests.
• to catch formatting errors, use autoformatters.
• to catch common mistakes, use linters.
• to catch common security problems, use a security scanner.

Don’t blame reviewers for missing these things.

Code review should not be how you catch bugs.

What Is Code Review For, Then?

Code review is for three things.

First, code review is for catching process failures. If a reviewer has noticed a few bugs of the same type in code review, that’s a sign that that type of bug is probably getting through review more often than it’s getting caught. Which means it’s time to figure out a way to deploy a tool or a test into CI that will reliably prevent that class of error, without requiring reviewers to be vigilant to it any more.

Second — and this is actually its more important purpose — code review is a tool for acculturation. Even if you already have good tools, good processes, and good documentation, new members of the team won’t necessarily know about those things. Code review is an opportunity for older members of the team to introduce newer ones to existing tools, patterns, or areas of responsibility. If you’re building an observer pattern, you might not realize that the codebase you’re working in already has an existing idiom for doing that, so you wouldn’t even think to search for it, but someone else who has worked more with the code might know about it and help you avoid repetition.

You will notice that I carefully avoided saying “junior” or “senior” in that paragraph. Sometimes the newer team member is actually more senior. But also, the acculturation goes both ways. This is the third thing that code review is for: disrupting your team’s culture and avoiding stagnation. If you have new talent, a fresh perspective can also be an extremely valuable tool for building a healthy culture. If you’re new to a team and trying to build something with an observer pattern, and this codebase has no tools for that, but your last job did, and it used one from an open source library, that is a good thing to point out in a review as well. It’s an opportunity to spot areas for improvement to culture, as much as it is to spot areas for improvement to process.

Thus, code review should be as hierarchically flat as possible. If the goal of code review were to spot bugs, it would make sense to reserve the ability to review code to only the most senior, detail-oriented, rigorous engineers in the organization. But most teams already know that that’s a recipe for brittleness, stagnation and bottlenecks. Thus, even though we know that not everyone on the team will be equally good at spotting bugs, it is very common in most teams to allow anyone past some fairly low minimum seniority bar to do reviews, often as low as “everyone on the team who has finished onboarding”.

Oops, Surprise, This Post Is Actually About LLMs Again

Sigh. I’m as disappointed as you are, but there are no two ways about it: LLM code generators are everywhere now, and we need to talk about how to deal with them. Thus, an important corollary of this understanding that code review is a social activity, is that LLMs are not social actors, thus you cannot rely on code review to inspect their output.

My own personal preference would be to eschew their use entirely, but in the spirit of harm reduction, if you’re going to use LLMs to generate code, you need to remember the ways in which LLMs are not like human beings.

When you relate to a human colleague, you will expect that:

1. you can make decisions about what to focus on based on their level of experience and areas of expertise to know what problems to focus on; from a late-career colleague you might be looking for bad habits held over from legacy programming languages; from an earlier-career colleague you might be focused more on logical test-coverage gaps,
2. and, they will learn from repeated interactions so that you can gradually focus less on a specific type of problem once you have seen that they’ve learned how to address it,

With an LLM, by contrast, while errors can certainly be biased a bit by the prompt from the engineer and pre-prompts that might exist in the repository, the types of errors that the LLM will make are somewhat more uniformly distributed across the experience range.

You will still find supposedly extremely sophisticated LLMs making extremely common mistakes, specifically because they are common, and thus appear frequently in the training data.

The LLM also can’t really learn. An intuitive response to this problem is to simply continue adding more and more instructions to its pre-prompt, treating that text file as its “memory”, but that just doesn’t work, and probably never will. The problem — “context rot” is somewhat fundamental to the nature of the technology.

Thus, code-generators must be treated more adversarially than you would a human code review partner. When you notice it making errors, you always have to add tests to a mechanical, deterministic harness that will evaluates the code, because the LLM cannot meaningfully learn from its mistakes outside a very small context window in the way that a human would, so giving it feedback is unhelpful. Asking it to just generate the code again still requires you to review it all again, and as we have previously learned, you, a human, cannot review more than 400 lines at once.

To Sum Up

Code review is a social process, and you should treat it as such. When you’re reviewing code from humans, share knowledge and encouragement as much as you share bugs or unmet technical requirements.

If you must reviewing code from an LLM, strengthen your automated code-quality verification tooling and make sure that its agentic loop will fail on its own when those quality checks fail immediately next time. Do not fall into the trap of appealing to its feelings, knowledge, or experience, because it doesn’t have any of those things.

But for both humans and LLMs, do not fall into the trap of thinking that your code review process is catching your bugs. That’s not its job.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor!