Fifth post in the AI gateway series. Before: why it exists, how to build it, what it costs in latency and where local models fit.
Summary
- A real customer-interaction analysis pipeline — calls and chats, in Portuguese — running on a proprietary LLM API. It works, and it charges three prices.
- I distilled a 9B open model from the production system itself, using 5,000 examples. The 18,000-token prompt shrank to about 400.
- On the verifiable task, the student is statistically indistinguishable from the teacher against an independent human gold standard.
- 36× lower cost, computed from a real invoice and metered GPU-hours — not from a price sheet.
- The saving doesn't come from cheap GPUs. It comes from schedulable ones — which changes who can repeat the result.
1. The three prices of a proprietary API
The system worked. That matters — this was not a rescue project, it was a production pipeline delivering interaction analysis.
But it charged three prices that rarely show up in the same conversation:
A bill tied to the vendor. Every new unit of volume is variable cost, and someone else owns the price.
Latency and availability outside your control. When the provider degrades, you find out alongside your users.
Personal data leaving continuously to a third party. Customer interactions are customer conversations. Every call is a transfer, and data protection law has an opinion about that.
That third price changes the nature of the decision. The first two are optimisable; the third is structural, and it is what I wrote about in Your AI usage policy doesn't prevent the leak.
The question was direct: how much of this system's quality can be preserved by distilling a small open model running on your own infrastructure?
2. Pick the base model with the production prompt
First methodological decision, and the one that saved the most time later: I evaluated 13 open models under the same production prompt, 18,000 tokens long. No generic benchmark.
Public leaderboards measure average performance on average tasks. What decides here is performance on your task, with your prompt, in your language — and the ordering changes when you change the criterion. A model that leads a table can fall apart on a long Portuguese prompt with a rigid output format.
If you repeat this: run the selection with the prompt already in production, before training anything. It is cheap and it eliminates half the candidates.
3. The distillation
5,000 examples, generated by the production system itself. The production system is the teacher: over time it has already produced the input-output pairs you need. Nothing had to be labelled from scratch.
That is a practical consequence of the argument in the local-model post: the training set is a by-product of having run the large model with tracing on. Whoever instrumented the call from the start already has the dataset; whoever didn't will have to wait for it to accumulate.
Supervised fine-tuning, 9B open model.
The part I didn't expect
The 18,000-token prompt shrank to about 400.
The entire project configuration — classification rules, output format, edge cases, domain vocabulary — stopped being an instruction repeated on every call and moved into the student's weights.
That has a direct effect on the bill, and it's a large part of the 36×: you don't pay 18,000 input tokens on every request. But it has a side effect worth knowing: configuration that was readable in a prompt file is now implicit in the model. Changing a rule stopped being a text edit and became a re-distillation. It's a trade, not a pure gain.
4. The evaluation protocol
This took the most work, and it is what I would defend first if someone challenged the result.
A quality number without a protocol is worthless, because whoever produced the model gets to pick the metric that flatters it. So: three independent instruments.
An automatic metric, on the part of the task that is verifiable.
A panel of LLM judges from vendors different from the teacher's. This matters: a judge from the teacher's own family tends to prefer the teacher's output. Using distinct families removes part of that bias.
A blind human annotator, with no knowledge of which output came from which system.
5. What the three instruments said
On the verifiable task — binary Yes/No scoring — the distilled student reached 87% agreement with the teacher.
And the test that matters more: against an independent human gold standard, the student is statistically indistinguishable from the teacher. Not "almost as good" — the difference between the two does not survive testing at the available sample size.
On the open-ended fields, under blind comparison, judges did not separate the 9B model's analysis from the production system's in 92% of cases.
On cost, 36× lower — computed from a real invoice and metered GPU-hours, not from a price sheet multiplied by an estimate.
6. Two lessons worth more than the numbers
6.1 The saving isn't that the GPU is cheap. It's that it's schedulable.
This is what people most often get wrong in the arithmetic.
A whole month of inference fitted into roughly 11 GPU-hours. The load is bursty: the pipeline processes batches of interactions, not continuous traffic.
The same GPU running 24/7 would have cost more than the API. The gain didn't come from cheap hardware — it came from only turning the machine on when there is work.
The practical consequence contradicts the common mental model: if your load is interactive and constant, the local-model arithmetic is far worse than in this case. Before projecting savings, look at your load profile. Bursty schedules; interactive doesn't.
6.2 Evaluation is worth more than training
The effort in this work was not the fine-tuning. Fine-tuning a 9B on 5,000 examples is a matter of hours.
The effort was building a defensible comparison protocol: the same prompt for everyone, judges from families other than the teacher's, blind human validation, and a power analysis to know what the sample can and cannot claim.
That last part almost never appears. Without a power analysis, "the student matched the teacher" can mean either "they are equivalent" or "the sample is too small to detect the difference". Those are opposite conclusions, and only the calculation separates them.
If you do this at your company and have to cut scope, cut the training, not the evaluation. A worse model with reliable measurement is a decision; a better model with loose measurement is a bet.
7. The deliverable isn't the model
The final product is not the weights file. Weights age — the domain shifts, the product changes, a better base model ships.
The deliverable is the loop: the company can re-distil from new traffic, re-evaluate against the same protocol, and promote a new version on automatic criteria rather than opinion.
It's the same idea as the task alias in the gateway: which model serves which task becomes reviewable configuration, not a choice buried in code. The distilled model becomes the destination of the local alias, and switching versions is one line.
8. Who this is for
If your load is interactive and constant, redo the arithmetic with your own profile first. This case's 36× comes from bursty load, and it does not transfer.
If your task is open-ended — free-form writing, long reasoning, synthesis over a lot of context — a distilled 9B will not match a frontier model. This result is from a narrow, well-defined task, which is where distillation works.
If your problem is personal data leaving the company, the conversation changes character: even with no cost gain, a local model is the only option that turns a contractual guarantee into a technical one.
And before any of that, step zero is the same: knowing which tasks you have and how much each one runs.
How we help teams adopt AI
We work with engineering teams putting AI into the real development workflow — not as an experiment, but as an actual part of how the team ships.
That means choosing the right tools for the team's context, configuring them in a way that makes sense for the company's data policy, and making sure developers know how to use them in a way that genuinely increases productivity instead of adding friction.
If you have a development team and you're trying to put AI to work seriously, get in touch.