AI Agents: Experience Doesn’t Fit in a Prompt

On a recent project, our team experimented with spec-driven development using AI agents. We expected to spend more time writing specifications and documenting project decisions. We also expected to review more generated code than we normally would.

What I wasn’t prepared for was how much time we spent writing down things our team had never really needed to write down before.

Ask a group of developers what they enjoy least about software development. Documentation and code review usually come up somewhere on the list. Our workflow suddenly involved a lot more of both. Every story included implementation work, but it also meant refining specifications, updating project skills, reviewing Markdown files, and ensuring all of that context still matched how we wanted the project to evolve.

The effort wasn’t a complete waste; as the project matured, the agents became much more consistent. The context we added early in the project did often pay off.

It also introduced a different kind of maintenance work.

Experience doesn’t fit in a prompt.

One of the first instructions we gave the agents was to follow test-driven development. That worked well until the stories became smaller.

A simple UI change often came back with several new tests and updates to the surrounding specifications. The implementation was correct, but the amount of supporting work didn’t match the size of the change.

An experienced developer knows when a small visual adjustment probably doesn’t need another round of tests. That’s usually something you pick up through code reviews, pairing sessions, and working with the rest of the team. We hadn’t explained any of that.

Eventually we stopped relying heavily on broad instruction and started adding examples instead. We documented the kinds of changes that should include new tests, the kinds that shouldn’t, and the testing patterns we wanted repeated. We also pointed to examples that already existed in the codebase. This all led to a much better outcome for agentic development.

We started making decisions earlier.

The same thing happened outside of testing.

Before asking an agent to implement a story, we found ourselves spending more time talking through service boundaries, naming, and data models as a team. Those conversations weren’t new. What was new was the external artifacts that were produced from those conversations.

For example, we started drawing simple diagrams before implementation and checking them into the repository. We added project-specific guidance alongside Angular’s published skills. When we found a pattern we liked, we preferred pointing to an existing implementation instead of describing it again in another prompt.

None of those changes necessarily made the project more interesting or understandable for the developers on the team, but they did make the agents much more predictable.

More context isn’t always better.

As the project continued, the repository accumulated more context alongside the source code.

Some of it had a clear purpose. The Angular skill established framework conventions. Figma designs preserved design intent. Code examples gave the agents patterns to follow instead of inventing new ones.

Other additions were harder to organize.

Every time we learned something new, we had another decision to make. Should this live in a project skill? Should it become part of the specification? Is it better as an implementation example? Is it something we should just expect developers to know?

Over time, the collection of skills started reminding me of a deck-building game. Early on, every new card feels useful. Eventually, your deck becomes large enough that individual cards stop feeling important.

Project skills felt similar. Each one solved a real problem. Together, they became harder to navigate. We started spending more time deciding where guidance belonged and whether an existing skill should grow or a new one should exist.

That felt surprisingly similar to designing software. Clear responsibilities were easier to maintain than one large collection of overlapping guidance.

Some documentation serves a different purpose.

One thing I kept coming back to was that not every Markdown file was written for the same audience. Architecture notes, domain models, and business rules helped developers understand the project. Project skills and implementation examples were there to shape future implementations. These artifacts naturally read differently.

For a while, I caught myself reviewing every generated markdown change with the same attention I would give an architecture document. That wasn’t sustainable. I learned to draw a line, and identify what context was there for the agent and what was a valuable investment for developers’ attention.

While I am still learning just where to draw that line, what I do know is that we were asking one collection of documents to do two different jobs.

Looking Back

One of the biggest changes on this project wasn’t learning how to work with AI agents.

It was discovering how much of our team’s experience had never needed to be written down before.

That experience used to spread through conversations, pairing sessions, and code reviews. Working with agents pushed more of it into the repository.

I’m not convinced every project needs that level of investment. On a short engagement, there were times when maintaining the growing collection of specs, skills, and examples felt like as much work as implementing the feature itself. Even so, our team’s experience is worth preserving – I don’t think I ever would go back to starting from a blank slate.

Conversation

Join the conversation

Your email address will not be published. Required fields are marked *