The LLM Is Not a Junior Engineer

Key Highlights:

License is not valid, please check your API Key!

---

Rewrite the following article in a natural, human-like tone. Keep the meaning the same but improve clarity, structure, and readability. Do NOT mention any source, website, or external reference. Return clean HTML paragraphs:

In the wake of my last essay on why I don’t vibe
code, I heard from various people on the
Internet who read it (or the Bluesky skeets that inspired it). Some had adopted
similar stances on their own, often for overlapping reasons. Others were
dedicated vibe coders who wanted to share what practices they used that made
things manageable for them. I appreciate the feedback! I’m not going to change
my stance, but it’s good to get perspective on how actual developers (and not
senior executives or corporate marketing materials) have engaged with this
technology.

Contrary to how I might seem at times, I do think there could be a place for
LLM models in modern software development. I do understand why
some developers are entranced with the ability to create applications in hours
and turn prototypes into products. I see how it could be highly useful for home
hobbyists, back-office bureaucrats and expert engineers to build the low-stakes software
they’ve never had the time or skill for.

But I think it’s also incredibly dangerous to assume these same benefits apply
equally for large software engineering teams or more sensitive software
applications. It might be one thing if your vibe-coded personal recipe app
mangles a measurement conversion; it’s entirely different if it destroys your
production
database
(even if it pretends to be really sorry afterwards). It’s one thing if you stay
up late coding on your personal project (I can’t
criticize), but another if your team is just merging in
code without testing or reviews because you’re too afraid to be the person who
is slowing the team down. And I haven’t seen many public example where teams have set
explicit bounds on how and where they want to use AI.

I haven’t been able to find many good examples of how teams are using Generative
AI (GenAI) effectively. Much of the literature is still very much in the early
phase of the AI lifecycle, with many articles exulting more in how AI replaces
the need for software teams rather than
how it might boost them. Perhaps, there are people figuring it out, but it feels
like we’re going to continue to see a new land-speed records in
self-owning
before we learn how to do it right.

And that’s probably how it will be. Software development as a practice has been
informed by decades of experience of how to do things terribly wrong, and many
of the standard practices we follow today were learned the hard way. Maybe we
need a few more years of companies deleting their entire code base or shipping
with critical errors (perhaps with a few high-profile outages or bankruptcies in
the mix) before the industry figures out how to sustainably work with these new
tools. These are some of the things I’ve been thinking about though.

The LLM is Not a Junior Engineer

First, I need to get something off my chest. It’s fairly common in our industry
to anthropomorphize GenAI products and describe them as junior engineers or
similar low-level coworkers. Stop it! While it may be useful to think of LLMs
as interns instead of
gods,
this framing still grants AI a conceptual personhood that makes it seem more
capable and reliable than it actually is. And it’s highly insulting to the
actual junior engineers in the industry, who are usually some of the most
talented and hard-working individuals you will find.

An AI model is not a person or even sentient. It has no long-term memory. It has
no internalized morality of what are good or bad behaviors (apart from what
might be implicitly reflected in its training data and reinforced in
post-training calibration). It doesn’t learn from any of its actions itself.
Instead, developers make it “learn” by carefully crafting introductory texts
which tell the LLM how to act and what things to avoid. And it seems to work
mostly, as long as someone remembers to tell the AI to stop talking about
goblins so
much.
It is indeed impressive and little magical that it does work so well most of the
time, but there are no
guarantees
that it won’t go wrong either. To help the LLM build on previous steps, many
coding agents will write to and read from a working memory file. This memory
itself is also included as part of the inputs into the model for each new step,
which means the longer the LLM is used on a single problem, the slower and more
expensive each successive query gets, to the point where some engineers have
reported hitting their weekly usage quotas within a single day. And when the LLM
finally fills out its limited context window, all sorts of wrong things will
happen, from API errors to selective amnesia as well as “lost-in-the-middle”
confusion and issues inferring responses to new prompts. To mitigate this, some
agentic models will include processes to summarize and compact their own
memories; this is a lossy compression by its very nature, so there is some risk
of distortion and loss there. Others will regularly just start over with new
agents that can stumble into the same mistakes and suggestions as their
predecessors without active intervention.

All of which is to say, if an LLM were a person (to be clear, he is not.), he
would be an absolute nightmare to work with. Every LLM agent is essentially a
combination of Amelia Bedelia
and Leonard Shelby from
Memento. It is up to you to
tell him precisely what he should or should not do. And it is also your
responsibility to keep a written memory that he will need to reference
constantly to know how to make sense of his world. Also, he doesn’t know or care
about anything else outside of his written notes, and you will feel intense
pressure to keep his notes as concise as possible without messing up. And when
you do mess up, other developers will blame you and not him. Your corporate
culture and values? Your general approach to architecture and testing? Your
long-term product road map? He doesn’t know it and doesn’t care. He doesn’t care
about how well the company is doing financially or its biggest challenges. He
will face no legal liability or professional consequences when he screws up
something. You know nothing about his background before he walked into the door
(and the secret instructions that his temp agency handed him before that so he
wouldn’t talk about the goblins). You do know he sometimes lies about things,
but it’s not always clear how much he’s lying beyond the very obvious whoppers
you’ve already caught. He can make recommendations and write code, but he has no
idea where it came from and how easy it is to support. He acts like a friend,
but at his core, he has no loyalty or empathy. He has no shared experiences with
you or fun weekend plans. He’s a perfectly impenetrable black box.

Junior engineers do learn, however. And unlike the hypothetical LLM-as-a-person,
they arrive at your company laden with context on how to work with your team.
They’ve grown up in a culture, learned the rules of society, have probably gone
to school for many years and likely learned some important things there. They
belong to families and possibly have heir own. They’ve put down roots in the
community. To work at your company, they’ve provided a resume and have gone
through multiple stages of your interview process. They’ve worked hard to land
this job and they want to keep it! They know that malicious behavior or neglect
will lead to real and painful consequences – being fired, litigation or even
criminal charges – that could potentially derail their entire careers. And even
without considering the threat of consequences or possibility reward, they will
still strive to do things the best they can. Because it just matters to them
in a way they might not always fully articulate but feel innately.

Of course, junior engineers do not move as fast as AI models, because they do
not come prepackaged with knowledge and must live in a world more complicated
than a stream of data. It is true that junior engineers will often make
mistakes, but they also generally learn from those too. And yes, genuinely
malicious engineers can cause real damage of their own as “insider
threats,” but there is at least
the possibility of consequences – being fired, litigation or even criminal
prosecution – including the fear of never being able to work in the industry
again. I don’t claim that junior engineers are perfect or infallible, but I do
believe that it’s also unfair to compare the worst of junior engineers to the
best of Generative AI models. And the most junior engineers are simply
phenomenal.

Best of all, junior engineers will mature with time into highly competent senior
engineers, engineering managers, directors and architects. But the GenAI models
will simply be replaced with new black boxes. Your investment of time and money
in them will earn dividends as they become more capable members of your staff.

‘Magic’ and ‘More Magic’

Junior Engineers will also rarely embarrass you.

In a recent profile of vibe coders, the New York Times included a tip from one
developer to tell his agent to not do things that would be
embarrassing,
leading the author to wonder how that would even work:

Embarrassing? Did that actually help, I wondered, telling the A.I. not to
“embarrass” you? Ebert grinned sheepishly. He couldn’t prove it, but prompts
like that seem to have slightly improved Claude’s performance.

This could of course be another fine example of pranking the
Times, but I’ve seen enough
similar guidance from other developers to believe this advice is genuine. That
doesn’t make it real however. Despite many claims about their effectiveness,
these prompting instructions are often little more than “prompting
folklore” that seem to work, but we lack an
explanation of why they would work or if any particular prompts are more
effective (maybe asking please would help? Or making threats to a machine that
will never truly feel threatened?). In some cases, other helpful prompting
strategies like asking the model to pretend to be an expert might improve the
odds of the AI agreeing with you, but actually damage its
accuracy. My point here is that it is going
to be a while before we understand what prompts and usage patterns are effective
and even longer before we understand why they work. How much should we
continue to share this superstition as useful?

I am reminded of an anecdote in the Jargon File about a
“magic” switch at the MIT AI
lab. As the story begins, an
early programmer (or “hacker” in MIT
parlance) was roaming the
halls of the AI lab when they noticed a curious switch bolted on to a
PDP-10:

You don’t touch an unknown switch on a computer without knowing what it does,
because you might crash the computer. The switch was labeled in a most
unhelpful way. It had two positions, and scrawled in pencil on the metal
switch body were the words ‘magic’ and ‘more magic’. The switch was in the
‘more magic’ position.

I called another hacker over to look at it. He had never seen the switch
before either. Closer examination revealed that the switch had only one wire
running to it! The other end of the wire did disappear into the maze of wires
inside the computer, but it’s a basic fact of electricity that a switch can’t
do anything unless there are two wires connected to it. This switch had a wire
connected on one side and no wire on its other side.

It was clear that this switch was someone’s idea of a silly joke. Convinced by
our reasoning that the switch was inoperative, we flipped it. The computer
instantly crashed.

Imagine our utter astonishment. We wrote it off as coincidence, but
nevertheless restored the switch to the ‘more magic’ position before reviving
the computer.

The author related the story to someone else a year later who insisted on going
back to look at the switch. They flipped it and the machine crashed yet again.
Obviously, this switch wasn’t actually controlling magic inside the computer,
but it was doing something, even if there were no satisfying actual explanations
of what that might be.

Not everything in the world needs an explanation, and many of us have certain
superstitions that let us pretend that we have some control over the arbitrary
randomness of the universe. A superstition like the gambler blowing on his dice
before shooting craps is harmless enough until the gambler starts to really
believe that his superstitions are real and that failing to follow them will
result in disastrous consequences. This phenomenon is called the illusion of
control, and it is common
for people to feel this when faced with completely random processes.

Large Language Models (LLMs) are not as simple as a game of craps, but they are
random at their core. Instead of a game of craps, we could consider an LLM as an
inordinately complicated pachinko
machine that
uses snippets of text instead of ricocheting ball bearings. Most LLMs include a
temperature parameter that injects a specific amount of randomness into how
the LLM selects its next generative response because this can jostle the model’s
completions onto a different pathway and make it seem more creative. This is
great for open-ended conversations, but not ideal for tools; I don’t imagine
you’d like to use a hammer that every so often would spray you in the face with
confetti. But, even when the temperature is set to 0, LLMs remain innately
nondetermistic
because of how floating point operations work. And, of course, companies can
also tweak the operating parameters for their models at any time, which could
have its own unexpected effects on model behavior for your applications (hey,
have you heard about the goblins?).

As a programmer, this nondeterminism makes me queasy. Suppose that I give an LLM
a prompt, notice an obvious error, rephrase my prompt and get a second correct
response in turn. In a purely deterministic model, I can be certain that my
modification to the prompt was the thing that fixed the model’s response because
it was the only thing that changed. Software is generally deterministic because
it allows us to understand things as cause and effect. Flipping the switch away
from “more magic” always causes a crash. It also lets us repeat the
circumstances that cause bugs to occur (and know that our fixes work). For
instance, a certain LLM model might always miscount the letter ‘r’ in
strawberry because it uses a deterministic
model for tokenization.

Under the nondeterministic model, I can no longer be sure. Maybe my change to
the system’s input did fix its output. But it’s also possible that what simply
happened is that the odds of the LLM producing erroneous output that I would
notice twice in a row were just suitably low. It’s possible that my prompt
modification had perhaps only a subtle effect on the system’s behavior or maybe
even no effect at all. Was it real or a
Heisenbug? When the system deletes my
production DB and backups, is it because I have a skill issue or was I just
particularly unlucky? I’ll never know.

Of course, we can build systems to handle unpredictability and bad luck. One
common approach is to use multiple computations and then check them against each
other. Indeed, one way to reduce the risks of individual LLM errors is to ask
the same question multiple times against multiple models or use judges to assess
output. The challenge is that this approach will both dramatically increase
costs as well as reduce the responsiveness of LLM models in specific
applications. It reminds me, in a way, of error correction code (ECC) memory
that protects against the relatively rare soft error bit flips caused by cosmic
rays This technology has been around
since the 1980s but it is not everywhere, because those memory correction
techniques increase costs and decrease system performance. And so, it makes
sense for a computer system controlling a nuclear reactor, but it might be
something you might not need for a graphics processing
unit
that normally would just power video games. It goes without saying that the risk
of LLM-induced failures is exponentially higher than those caused by an errant
cosmic ray, but many developers are still stumbling in the dark on the best ways
to handle error correction and when to use it. Maybe we can start by accurately
thinking about risks.

Clarifying Risks

As an experienced software engineer, you learn to always anticipate that danger
can happen. Instead of dissolving into a puddle of
anxiety, a software development team learn to manage the fear and quantify
uncertainties into a document that outlines all the important risks. A specific
risk could be something technical (“an entire data center that we are using for
our cloud provider goes
offline”)
or more broadly existential (“we miss our launch deadline”), and it will often
include things that are not entirely in the team’s control. For every risk, the
team will determine if there is a suitable mitigation that allows the team to
recover from the risk (ideally, the best mitigation will prevent the risk
entirely) and if the team “owns” the risk mitigation and response. It’s also
highly useful to estimate the likelihood and impact of a given risk. Typically,
these are both estimated on a scale from 1 to 5 and multiplied to generate an
overall severity estimate – under this approach, low-impact but highly common
risk might turn out to have a higher severity than a more dangerous risk that is
extremely rare. Severity gives teams the ability to prioritize which risks to
address first and calibrate how much they should worry about specific problems.
To avoid being overwhelmed by a world filled with danger, teams will also
generally limit the number of risks they track at a given moment (usually to 20
or fewer) and will regularly review the list to add or remove items or
recalculate their severity.

There is an art to this approach, of course, and its estimates are more of a
heuristic than a precise calculation of what dangers might occur. And that’s
fine. The main point of the exercise is the process where the entire team
discusses the risk and not the just the risk register that they create. Everyone
contributes to the discussion of what risks should be tracked and how to point
them, and it benefits immensely from multiple perspectives that can look at the
technical stack with a critical eye. It is also is informed by decades of
failures and fixes in how software is built and deployed to infrastructure.

I don’t know how to properly consider risks for GenAI usage. I think the
industry is still figuring out the failure patterns and mitigations for them. I
have observed that developers (and other team members) vary wildly in how they
assess the impact and severity of AI-related risks (boosters are excited and
skeptics are cautious), and I believe that this makes it harder for teams to
come to a realistic consensus of risks in AI usage in their products and
processes. I’ve also noticed that much of the existing discussion about GenAI
risks has been at a relatively high-level (e.g., “should we use this for
purpose X or industry Y?”) and it’s harder to find more granular examples
exploring the risks and benefits of using AI for very specific purposes in your
very specific context (e.g., “should we use this LLM model to generate synthetic
data for our testing environment?”). Many AI users and companies are actively
learning how these tools can go wrong and what to do about it, but this is a
case where risk assessments should skew pessimistically while they figure it
out. Maybe we should make a regular practice of asking “what is the worst that
can happen?” and identifying scenarios where “it gets worse than that” if we
don’t feel confident we’re pessimistic enough.

Human in the Loop

Last year, Amazon’s retail operations suffered several outages that directly
caused by the Kiro LLM tool. Amazon angrily retorted that the news reports were
wrong and the outages were actually caused by solitary developers making
changes under the advice of
AI.
That’s not much more reassuring. As I’ve mentioned above, we’ve had decades of
mistakes that inform modern DevOps practices for maintaining software
architecture. This includes best practices like “don’t let a single engineer
make tweaks to production” or “code changes should be reviewed by at least one
other engineer before being merged” or even “people (and the AI models using
their credentials) should have only the minimum amount of access
privileges they
need for their job.” It’s common for companies to ritualistically invoke the
phrase “human in the loop” to answer concerns about LLM safety. But how
effective is that guarantee?

Much has already been written about how extensive GenAI usage can create not
just cognitive displacement but even “cognitive
surrender”,
where people defer unthinkingly to the judgment of the LLM. Some go willingly,
but many are being pushed into it by management pressures. It seems like no
coincidence that the same Amazon retail division that has suffered several
AI-related outages is also heavily monitoring GenAI usage among
staff and setting ambitiously high targets for
adoption:

The effort calls for more than 2,100 engineering teams in the retail arm to
triple software code release velocity using what Amazon calls “AI-native”
practices, while a smaller group of at least 25 teams is expected to boost
output tenfold this year. Progress against these goals is closely tracked by
Amazon’s senior leadership team, known as the S-Team, according to the
document.

This model of setting targets before considering their effectiveness is how you
repeatedly take down production and create problems that are hard to unwind.
This not only creates cognitive surrender but also a moral abdication, where
engineering teams abandon their sense of duty to a vague hope that the AI won’t
err too badly. These metrics are very specific about what tools teams should use
and how they should act, but they don’t consider how this might affect the
product that the teams are working on (in this case, Amazon’s retail operations
and website) and which might be negatively impacted by this change. I will talk
more about the absurdity of these velocity metrics below, but I wanted to note
here how it is simply impossible for a “human in the loop” to act meaningfully
under these circumstances: direct action against GenAI usage that might stop
bugs will be penalized for making your team miss the target, but letting stuff
through that might take down production can be conveniently blamed on the AI (or
the single engineer we can scapegoat for using it). And so, the teams will hit
their targets, the executives will chalk this up as a success, and the website will just start
acting more erratically because that’s not as high a priority.

I think it’s essential for teams and organizations to firmly define what they
mean by a “human in the loop” and stick to that definition. Here’s a possible
place to start: any person on the team can hit the imaginary big red button that
stops the assembly line at any time for any reason (no shaming nor penalties
allowed). Never replace a person with an AI model, especially if that person
contributes necessary friction (product managers, accessibility testers,
security compliance); if you must, instead work with that role on ways that LLMs
might supplement their activities. Explicitly ask at team retrospective meetings
if people felt pressured or rushed into approving AI-generated work. Audit all
AI usage to ensure there are no places where an AI model is taking both sides of
a working relationship for quality control. For instance, an AI model should
never be allowed to write both code and the tests for that code, or to write
both design documents and implementations. These restrictions should be enforced
even if we are using two different agents of the same Large Language Model or
two different AI products to evaluate the model. Define the line and hold firm
to it.

And it may be that the line is not to us AI at all. Perhaps, you work in an area
where the legal or ethical risks are too great. Perhaps, you don’t want to help
support AI as it currently exists, where the choice is among several different
large organizations with terrible environmental practices, huge influence over
certain swaths of society and decidedly flexible approaches to morality. I feel
the same way. This isn’t politicizing AI, but it is recognizing that how you use
AI is an inherently political choice (as is basically everything else), and your
team should be making that choice willingly rather than feeling coerced.

The goal here is to be intentional about using LLMs where they might fit best
while reducing exposure to the risks of cognitive surrender and professional
laxity. For me, a good breakdown would be to use LLMs for situations where there
is accidental complexity (something that can be solved by better tooling) vs.
problems with essential complexity (how to architecture precise software models
to reflect the messy nuances of reality). I do think AI models potentially show a lot of
promise for building tests and fuzz testing, synthetic data generation, static
analysis to identify bugs, code exploration and summarization. But, I also think
a person should always be checking their work.

The LLM Budget Bomb

I’ve already mentioned that I’m a
cheapskate, so admittedly I’m overly
sensitive to this, but it’s shocking to me how little discussion there is about
the inevitable substantial rate hikes that will hit LLM usage as soon as this
year.
Over the past few years, AI companies have been drastically discounting their
products in the hopes of increasing market-share and gaining advantages against
their rivals. At the same time, they have been dramatically expanding their
capital expenditures by building out new data centers. At some point, investors
are going to want to see returns, and some changes to AI pricing have already
started to show:

Everyone I spoke to had some version of this problem — their token usage has
gone up, so their usage-based billing cost has gone up, or the tier they were
on no longer has the same cap, and now they’re having to go to a more
expensive tier to try to keep the same amount of usage per month as part of
their flat rate.

Many personal AI users have been using tiered products that come with usage
quotas which obscure the true cost of their activities (unless you hit your
weekly limit). But as costs increase, even those users might find themselves
forced to move to higher tiers. For instance, Anthropic recently removed Claude
Code from its $20/month tier and made it exclusive to the $100/month tier or
higher. The
company reversed course after facing widespread outrage from developers suddenly
contemplating a 5x increase in their monthly AI bill, but I think an price
increase for this feature is inevitable. Similarly, GitHub Copilot has announced
it is moving all customers to a metered model based on token
usage,
and that model is how Enterprise users of all the major LLM platforms are
already paying for that usage. Under this model, it’s up to developers to track
and manage and budget their AI costs by tracking their usage and setting
budgets. The problem is that those costs are essentially impossible to budget
for.

Some of you might be wondering what these tokens actually are. Generative LLMs
work by taking inputs of text to generate more text which is then fed in as more
text into the LLM to generate more
text until either it hits some
sort of stop condition or character limit. A Large Language Model roughly works
in a similar fashion to text autocomplete, but instead of suggesting the most
likely next letter, it suggests the next possible token that is the most
semantically likely continuation (with a bit of randomness thrown in). It does
that by calculating the probability of every possible completion based on the
data it was trained with and then usually (this is where temperature comes into
play) picking the most likely one. Whole words would be the best unit of text for making
these semantic associations, but there are just too many of them (especially
when you factor in spelling mistakes, etc.) to make the models feasible.
Instead, a given LLM reads its input as a sequence of multi-character tokens
rather than characters – for instance, “strawberry” might be read as “str”
“awb” “erry” by one model and “straw” “berry” by another. Tokens are an
engineering compromise that invisibly shapes how each model sees the world. And
since they are the basic unit of processing, they are also naturally the basic
unit for any metered billing (or usage quotas for different tiers in an flat
pricing model). Costs scale linearly with token usage, but this token usage is
hard to predict.

One problem is that it’s impossible to know in advance how many tokens even a
single LLM query will actually consume. It is possible to set hard limits so
that a session will not churn indefinitely, but that limit essentially acts like
a proctor for an exam telling students to put their pencils down; the LLM is not
able to strategically analyze and plan against its token budget. Some guides
suggest handling this through prompts that tell the LLM to be more terse in its
responses, but this is just another example of prompting folklore.

It is also worth remembering that LLMs produce plausible output (usually the
most likely responses). That is not the same as correct output. Unfortunately,
many LLM techiques to increase accuracy – chain of thought
prompting or using another
LLM as a judge – will
themselves increase token usage, pushing teams into the dilemma of balancing the
risks of errors against increasing costs. For instance, a team might find that
it has to add guardrails in the form of regular expressions and analysis by
another LLM model to check the output of an AI chat agent that has been acting a
little too helpful to hackers. They probably didn’t consider those costs at
first.

And we shouldn’t forget that LLM companies still control two different means to
shift more token costs onto user. First, companies will set a fixed rate on
their per-token cost. For instance, Anthropic’s Claude Opus 4.6 model charge $5
per million input tokens and $25 per million output tokens, while the older and
less powerful 3.5 Sonnet model is $3/$15 per million tokens respectively. These
rates are not regulated and could change at any time. In addition, token usage
can vary. Anthropic recently announced an upgrade to Opus 4.7 to better handle
certain agentic coding tasks. The company guaranteed it would have the same
per-token cost as Opus 4.6, but users have still seen dramatically higher usage
costs because the new model has a different tokenizer implementation, leading to
much higher token utilization for identical queries. Anthropic itself has
conceded that some tasks might require 35% more tokens, but developers have
seen increases of up to 46% in token
usage
and corresponding costs. There will always be a trade-off between accuracy and
costs; at some point, a 1% improvement in model performance might not seem worth
if it leads to a 10% increase in costs. But, the LLM provider is the one making
that decision, while you are the one footing the bill. At some point, you might
need to decide if the benefits still outweigh the costs, but how do you figure
out the benefits anyway?

Productivity Metrics Are a Scam

As companies have moved to mandate LLM usage for their teams, they have
increasingly relied on metrics to track how their employees are using the tool
and to brag about how fast their development velocity has become. In the example
of Amazon from above, they have stated their expectations is for teams to
triple their software code release output, which could possibly mean they
either want to triple the number of times they deploy something to production or
triple the lines of code they write every week or triple the number of work
tasks they complete. Google’s CEO has recently bragged that their AI development
tools have led to a 10% increase in developer
velocity
that sounds impressive, but how do they even measure that? Investigating this
question led me down a rabbit hole. First, I waded through a lot of useless slop
explaining the importance of measuring developer productivity without providing
any direction on how. Eventually, an astute reader tipped me of to a DX article
about measuring LLM
impact
that clarified they were indeed measuring improvemtn in developer-hours to make
this declaration. Great! Now, how do we measure developer hours then?

And this is where I ran into a wall. DX is a commercial product for measuring
Developer Experience metrics, and that documentation is behind a
paywall. So, it’s possible they are actually measuring
developer time. This likely doesn’t mean measuring the overall time of
developers working in the office (since I doubt the Google CEO would be bragging
about developers leaving work almost an hour early each day. Instead, it likely
is a complicated melange of various automated surveillance observations
(corporations are a big market for spyware) that are blended at different
weights into deriving a measure of overall developer time for each feature being
developed. As always, it’s important to ask what gets counted and what doesn’t.
Does typing time count even though it’s just people typing comments into Stack
Overflow? Do meetings not count, even if they’re important to coordinate what
the team is building? And then it’s important to ask how AI changing team
behavior might possibly change the output of these statistics. For example, it’s
pretty clear that developers in teams that are heavily using LLMs have to shift
more time into waiting for responses and reviewing pull requests; this
translates to much less typing and more scrolling and staring at screens for
minutes. It’s possible that the LLM is really reducing the overall time per
feature, but it’s also possible that LLM usage is just skewing how the metric is
calculated without any real improvement to developer productivity. I simply
don’t have enough info to say, but I’m innately suspicious about a metric that
is conveniently obscure.

Of course, most places don’t have the time or energy to invest in a platform to
collect Developer Experience metrics and instead they just wing it with what
they have. Returning to Amazon’s productivity goals, I’m not sure what metrics
they mean to measure for this. Obviously, just tripling the lines of code is not
necessarily a sign of quality, and it’s also unclear to me why tripling the
number of deployments would be better, since I assume Amazon has already been
following modern DevOps practices for frequent
deployments. Instead, I assume they are
just committing the cardinal sin of agile development: using estimation points
as a metric. For those of you who don’t know what I
mean, many teams doing agile development grapple with the problem of estimating
how long the work will take by using a points-based approach. Under this model,
we divide the work into discrete tasks that can be assigned to individual
developers and then estimate the work it will take to do such tasks. So, fixing
a simple typo on an admin screen might be 1 point, while a complex redesign of a
testing harness might be 8 points (many teams use a Fibonacci-based
scale
of 1, 2, 3, 5, 8, 12, but it’s not essential). Once we have pointed all the
tickets, we can then use this to estimate how much work the team can fit into a
single sprint of 2 weeks (or usually 10 working days) by adding up the number of
points that the team was able to complete in a few prior sprints and deriving a
points-per-developer-day rolling average you can use to figure out a target for
an upcoming sprint with only 9 working days and one of the developers out on
vacation. It’s definitely still an approximation rather than an exact science,
but the goal here is for the team to be realistic about how much work it can
reasonably do in the next two weeks and defer remaining work for later sprints.
The problem is that the organizational leadership doesn’t want to know “are we
able to somewhat reliably estimate work for the next sprint?” What they care
about is “can we publish a statistic to show how our corporate bet on AI is
paying off?”

Absent a crystal ball and intensive measures to track productivity, companies
will try to get a “good enough” answer out of the team metrics they do have, and
points seem like a natural choice. After all, if a given team now is doing 33
points in a sprint with LLM tools and it was doing 30 points six months ago,
that’s a 10% velocity gain! Except for all the ways it which it goes wrong. For
starters, we should check that the team isn’t achieving this gain by skipping
important safeguards like code reviews or passing tests. Furthermore, team
practices for estimation can drift over time, as the team compositions changes
or developers become better about underestimating the work. And of course, teams
might actively cheat, if there are incentives to be top performers or penalties
meted out to laggards – one way to get a 10% boost might be just to point a
bunch of 1-point tickets to 2 points. Sure, it’ll make it harder to estimate the
work of future sprints in the short term, but eventually the rolling averages
will reflect the new normal. Simply put, points are just a terrible way to track
team velocity and productivity. They’re just so inconsistent, unreliable, easily
manipulated and totally uncalibrated, that I would hesitate about making broad
inferences from comparing totals a year apart. But, they do let you do easy
math, and that’s good enough for people who just need to say they had a 23.7%
improvement in developer speed.

To be fair, this is not a new problem that is unique to LLM-assisted teams.
Most metrics are bad, but the ones that measure
team productivity or speed have always been especially deficient, as the example
above shows. They are so tempting though! This is because they are super easy to
compute and directly responsive to changes, letting you cite an instant
improvement or reduction as a quick win for the next all-hands meeting. For
instance, a company can tweak a setting and directly measure “40% more customers
are interacting with our AI agent” just by counting the API calls from its
front-end web interface. More customer-AI interactions doesn’t necessarily mean
better interactions or happier customers however! And the metrics which reflect
the real quality measures – customer satisfaction, site reliability, revenue –
that dictate whether the business succeeds or fails are often lagging indicators
that take a while to manifest and are harder to diagnose. If a team is hitting
high velocity scores by not reviewing any LLM code before merging it, that will
eventually be reflected in a product quality metric like uptime showing that the
system is crashing more frequently. The truth will out, but sometimes it takes a
while; by the time the company realizes that the increase in AI chats is leading
to a 23% surge in subscription cancellations, the damage might already be locked
in.

The main points from this section? Most AI-related statistics are potentially
suspicious and you should be wary before citing any of them uncritically (even
for ones reporting damaging effects from LLM usage). It is easy to measure an
action, but much harder to measure the effects of an action or gauge its
quality.

What if the AI Goes Away?

There is a joke from the “Two Dozen and One Grayhounds” episode in the sixth
season of the Simpsons where technical difficulties force the local TV station
off the air and it airs this disclaimer:

Your cable TV is experiencing difficulties. Do not panic. Resist the urge to
read or talk to loved ones. Do not attempt sexual relations as years of TV
radiation exposure have left your genitals withered and useless.

Lying in bed with his wife, Chief Wiggum lifts up the sheets and looks before
uttering “Well, I’ll be damned.” I do sometimes wonder if a similar moment of
reckoning is coming for companies that have leaned heavily into LLM usage and
eliminated large numbers of its staff as redundant. Will they find themselves
trapped in a spending cycle, locked into a particular vendor? What happens if
that vendor goes bankrupt?

I don’t particularly think there will be a catastrophic moment where all AI
companies are destroyed, although I do think it’s likely one of the major
vendors will implode into bankruptcy, with profound effects on the economy and
the software industry. For many companies, their AI usage will not end with a
bang, but with a series of whimpers, when they need to make strategic decisions
about their budgets and usage of the technology. As risks go, this is a pretty
important one to prepare for.

If you are on a software team using AI in your processes or products, it might
be worth regularly asking yourself what it would mean if the LLM went away?
Would this be a critical failure that would doom your operations or something
that you can fix? Is the LLM being used for production products that would break
instantly or just for development practices like automated testing which can be
revised? How much is this usage locked in to a particular provider or model? If
you are using the LLM to generate code, how much do other developers understand
that code? You wouldn’t abide a situation where everything broke because only
one developer understood that code and she’s on vacation this week, are you
accepting that for AI-generated code?

What Next?

I wrote this essay because I wanted to work through some thoughts about how LLMs
might be incorporated into teams working on software, in the hope that such a
future is possible to achieve and a thing worth having. I hoped to see something
in this that would make me feel more comfortable about AI usage in the craft of
software development, as an alternative to the grim future that Silicon Valley
is intent on building
now.
Perhaps, a better AI future is possible. For instance, I’m particularly
intrigued by the idea that a smaller and slightly-less-accurate-but-still-good-enough Medium Language
Model (MLM) that can run on a server or even your laptop might just be good
enough for many teams, and it’s certainly a lot cheaper! Maybe we will enter an
era of open-source and more ethical choices for AI
tools. I’m not
convinced, but I sincerely hope I’m wrong on this. For all of our sakes.