At Atomic Object, we’re encouraged to prototype and dogfood internal projects. Over the last four months, I’ve been building a prototype to help solve a part of my job that I dislike. The first version used an LLM to extract and classify information from documents, then connect related pieces so humans and agents could retrieve the context they needed.
Prototype with the best model you can afford.
I built the first version entirely with frontier models. They gave me a high-quality ceiling, dependable structured output, and consistent service. That let me test the product idea instead of debugging whether the model could do the job.
It worked, but it was going to get expensive as usage grew and other teams experimented with the prototype. Prototype spend averaged $1.58 per day in July. From August 3 through August 14, it averaged $6.33 per day, about four times higher.
Quest for cheaper inference.
Hosted inference providers let me compare open-weight models through standard APIs without running GPUs myself. I could change the model with a parameter, and the per-token prices were much lower than the frontier APIs I had been using.
Finding cheaper models was easy. Finding one that could do the prototype’s actual work took the rest of the experiment. Industry benchmarks weren’t good predictors because the workflow had its own inputs, schema, prompts, and failure modes.
The goal of the experiment was to find out where cheaper models would break the workflow, then decide which problems were worth solving.
1. Define success before comparing models.
Before comparing models, I needed a working definition of “correct.”
The prototype had already produced about four months of outputs from sample and dogfood projects. I had Codex assemble a mix of routine cases, difficult edge cases, and ambiguous classifications, then build a small HTML review tool. I reviewed the examples and recorded what I believed the correct outputs should be. That became the gold set I used to score each model.
The frontier model’s previous outputs were a useful starting point, but I treated them as suggestions rather than truth. The review tool showed me the underlying material, source information, and the reasons the model gave for each decision. In several cases, I disagreed with the prototype’s original answer. Reviewing those disagreements myself gave me a benchmark I trusted.
With the eval set in place, I replayed the same material through each model and checked:
- Did it capture the useful information?
- Did it correctly attribute information to its source?
- Did it return the right amount of detail?
- Did it follow the output schema reliably?
- How long did it take, and how much did it cost?
“As good as the frontier model” sounds measurable until two good models return different but equally useful answers. Using the same prompts and schemas, GLM, DeepSeek, and Kimi produced very different tradeoffs.
2. Change one thing at a time.
Once I had a gold set, I tried to make the experiments boring. I froze the inputs, prompt, output schema, and request settings, then changed one variable. For the first model comparison, that variable was the model. When I started tuning prompts, I held the model and examples steady and changed one instruction at a time. Otherwise, I would have no idea what made a run better or worse.
I used agents to score each run and surface disagreements that needed my review. Rather than reacting to one odd answer, I looked for mistakes that repeated across several examples. Each pattern gave me something specific to try in the next prompt.
Choosing the next direction still required product judgment. I relied heavily on product sense, gut feeling, and spending time in the outputs myself. A model could score higher by capturing more information while still making the product worse by giving the user too much to review. I had to decide which kinds of mistakes I was comfortable putting in front of someone.
Once I had a direction, I gave an agent a cost cap and let it run four to six controlled passes. It made one prompt adjustment, replayed the same examples, and reported the result with supporting data. Then I went back into the outputs, decided whether we were moving in the right direction, and repeated.
3. Be willing to change the task.
Prompt tuning started to feel like whack-a-mole. I’d improve one quality dimension and watch another get worse. One version captured more useful information but also surfaced routine scheduling and personal details. Another cut down the noise but bundled separate ideas together and missed useful information. After enough loops, I stopped asking how to make the cheaper model reproduce the old workflow and started looking at the workflow itself.
The prototype asked models to do three kinds of work: extract information, classify it, and decide how it related to existing context. I had treated all three as requirements because the frontier model made the overall workflow seem workable. Evaluating each job separately gave me a chance to question that assumption.
Relationship classification created more review work than value. People had to untangle ambiguous model judgments before they could finish reviewing the extracted information. The better design was to move that work out of the main review flow and give the model a narrower job. That made the prompt easier to tune and removed a layer of required review from the main workflow.
When a cheaper model repeatedly fails, inspect both the prompt and the job you gave it. The job itself may be overbuilt.
Here’s what we ended up with.
In our final cross-project evaluation, the open-weight model fully or partially reproduced 86.7% of the topics people had accepted from the previous system. Each extraction averaged about three cents, 91.5% less than the comparable frontier-model calls.
If you’re considering the same move, collect a few dozen examples your current system has already handled. Include routine cases, hard edge cases, and outputs you’re unsure about. Run a cheaper model against them, review the disagreements yourself, and change one variable at a time.
From there, you can decide where to spend your effort: a better prompt, a different model, or a simpler job.