If you haven’t yet read The Decoherence Trap by Kristen Maevyn, I highly suggest you do, because in a way this is my extension of those great ideas. Maevyn describes how individually reasonable changes can accumulate into a system that becomes increasingly difficult to change. What I want to explore is how we’re losing the shared understanding that helps us recognize those changes in the first place.
In my own work, I’ve been trying to protect that shared understanding by making a point of having the conversations that a chatbot makes easy to put off. Talking things through with a teammate helps us both understand how the system is changing. The more I think about what’s been working in my daily practice, the more I wonder whether the way we’re using these tools is interfering with how we learn to understand and direct our own work.
The Conversations We’re Putting Off
I can’t help but feel nostalgia for the “good old days” (not so long ago) when scope seemed to matter more and PRs weren’t thousand-line cannonballs. When I hit a wall back then, I would have to go get help, even if that meant admitting I didn’t understand something as well as I should have. Now I can spend that time casting out educational guesses in the form of prompts.
But this convenient activity defers the human conversation that would give me better context. There is always another prompt to try, and following it can feel easier than explaining my confusion to a colleague. That conversation used to accomplish several things at once, though: it helped me solve the problem, let them know how the system was changing, and brought together our understanding. I cannot help but shake my head at how quickly we went from warning each other about knowledge siloing to essentially siloing everyone alone with a chatbot that has no real accountability for the team’s goals.
The strange dynamic now at play on agentic software teams is one where each person can advance their concept of the project rapidly, pulled along by the current of their LLM. Meanwhile, their peers may be making similar mental updates based on entirely different assumptions. It is only later, when those assumptions manifest as incompatible implementations, that everyone finds themselves wondering what happened. You can jump right into debugging, but if the underlying issue is a disjointed vision of the system, no amount of prompting will replace the need for an aligning conversation. This can happen between teammates, or between the team and the client whose expectations nobody thought to clarify. Although misalignment happened long before AI, the amount of code we can produce before noticing it has changed considerably.
We currently try to mitigate this issue by instructing our LLMs to read and maintain spec documentation. There is value in that, but these documents can easily become a dumping ground for muddy information that grows out of sync with the runtime code or the client’s expectations. Even a well-maintained spec cannot tell us whether everyone who read it understood the same thing. If we haven’t resolved an ambiguity ourselves, we shouldn’t assume that asking an LLM to interpret it will resolve it for us. We may spend considerable time walking back an implementation built around an assumption that sounded reasonable until we spoke to the people using the software. Having the context written down helps, but someone still has to do the work of making sense of it with everyone involved.
Whose Time Are We Saving?
This is why I’m suspicious of treating the time it takes to produce code as the measure of how much time we’ve saved. Some of the supposed speed gains may simply time-shift the burden of understanding into the future. A developer produces the implementation quickly, but a reviewer has to reconstruct its reasoning, or a maintainer has to discover its assumptions when something breaks. For those of us who make a living maintaining production systems, that future is somewhere we will actually have to reside. Of course, AI can save time, and there are tasks where delegating the implementation might make sense. But before celebrating the speed, we ought to ask whose time we saved and who will eventually have to understand what we produced.
I believe the danger of allowing this process to continue unchecked is cognitive surrender. There is only so much mental energy we have as people, and when we spend disproportionate amounts of it navigating something labyrinthine, handing the problem back to the chatbot becomes increasingly appealing. Maybe it will figure it out. But. The less we understand, the harder it becomes to question the next implementation or explain why it doesn’t belong. Eventually, the tool becomes something we depend on to work in a system we are still responsible for maintaining.
Understanding Is Part of the Work
It doesn’t have to be like this, and I think we should reconsider where in the development loop we are using LLMs. What I find useful is their ability to point out holes in an implementation, produce counterexamples, and answer questions across large real-world datasets. With that in mind, I want to explore a different style of development: have people write the implementation, and use LLMs to provide feedback. I want this because, when I’m able to work through the code myself, as a medium in my craft, its shape can challenge an assumption or draw my attention to a case I missed. Ultimately, I have to decide whether that case matters, what the system actually needs, and how the change should fit. Using LLMs in more of a discovery capacity seem like a promising way to apply the tool while remaining engaged in the craft of development.
Writing the implementation is often how I discover what I don’t understand. Choosing a name forces me to think about what something represents, and deciding where a feature belongs makes me consider how the system is organized. Awkward code may reveal that I need to ask a teammate why a rule exists, or ask the client what should happen in a situation we haven’t discussed. Working through those decisions gives me opportunities to build a shared understanding that I can’t as easily extract from generated code. That doesn’t mean typing every line guarantees understanding, or that an LLM can never build something sensible. It does mean that we currently bear the time-shifted burden of the learning that used to take place more slowly, along the way.
And look, the ability to generate code should not warrant a different standard for PR quality. Before, if I wrote a misleading variable name, that would be critiqued in review and I would have to change it to something sensible. If I constantly put up gigantic PRs, you can believe they would be rejected, and I would get an earful. Yet somehow, once an LLM is involved, there can be an expectation that everyone else should accommodate the volume of its output. If a change is too sprawling to explain or review properly, generating it quickly hasn’t relieved us of the responsibility to make it manageable. Having access to these tools shouldn’t lead to the passenger seat in this regard.
Becoming Intentional
What we need is to become more intentional about protecting the coherence of our projects, especially the understanding shared between the people working on them. When you hit a mental wall and find yourself staring at a confusing block of generated text, give yourself the space and time to understand what is being said. Push back on confusing phrasing, question the assumptions. And to whatever extent you can, reach out to someone else: pair on the difficult part, walk through the change together, or take it to a whiteboard.
These conversations give us a chance to discover where our understanding differs before we build those differences into the software. Perhaps we should judge AI assistance partly by whether it helps us do that work. I want to finish a project understanding more about the system than I did when I started, and with ideas I can explain to the people maintaining it alongside me. We are building software that works well, because we understand it.