93% cheaper. Nearly as accurate. Ships only when proven.
We ran Afterburner end to end on 2,084 real customer messages to a bank. A 135M model, trained on an 8-core M2 CPU, beat the frontier offline and cut cost per successful task by 93%. In the live A/B it came within 3.4 points of the frontier on messages it had never seen, inside the agreed margin. There wasn't yet enough traffic to prove that with 95% confidence, so the system held it back before anything ramped.
That's the point. A cheaper model only replaces your frontier calls after it proves, on your traffic and your outcomes, that it does the job as well. When it can't, nothing changes and you get a record of why.
The whole 53-minute run in 80 seconds, recorded from the live dashboard
See every call, what it costs, and which ones are worth moving.
The bank changed one setting, its client's base URL. The proxy logged every call, and the audit sorted the bank's two LLM call sites by what to do about them.
- ^The saving: the gap between what these calls would have cost on the frontier and what they cost to serve.
- ^Where each call went: our model, a fallback, or the frontier.
- ^The audit's verdict for each call site, with its reason.
- Spend traced to code
- 100%
- Routing call
- Migrate · 77% of spend
- Reply drafting (agent step)
- Frontier only · 23%
+−See the analytics pageDashboard · Audit report
Tie each call to the line of code that sent it.
A read-only scan found every place the bank's code calls an LLM and drafted a task card for each. Live prompts were fingerprinted and joined to that line.
- ^The highlighted line in bank_app.py: the routing call.
- ^The task card: 12 allowed labels, so every answer can be checked.
- ^Recent calls: who answered, and whether the support team kept or re-routed the answer.
- Call site
- bank_app.py:21
- Prompt templates joined
- 1 fingerprint · 2,384 calls
- Distillability
- 90 / 100
+−See the analytics pageTrace page
Train a small model on the answers the business kept.
Only frontier answers the support team did not re-route became training data, so the model never learned the frontier's mistakes. The newest messages were held back to replay offline.
- ^Offline, our model beat the frontier overall.
- ^By intent, it trailed on look-alike pairs such as compromised_card vs lost_or_stolen_card.
- ^Confidence vs correctness: the fallback threshold came out at zero, so nothing would fall back.
- Training data
- 808 kept answers
- Model · hardware · time
- 135M · M2 CPU · 45 min
- Offline: ours vs frontier
- 90.2% vs 85.8%
- Fallback threshold
- 0.00
+−See the analytics pageModel page
Let live traffic decide, not the offline score.
Half the bank's customers got our model, half the frontier, on 962 messages the model had never seen, at a steady three per second. Success was the bank's own outcome: the message stayed in the queue it was routed to. The margin was five points, not the two of a typical pilot, because only 962 unseen messages were available.
- ^Cost and latency both pass.
- ^The interval chart: the one-sided bound lands left of the 5-point margin, so quality isn't shown to be no worse.
- Success: frontier vs ours
- 89.5% vs 86.1%
- Difference, 95% CI
- −3.4 pts (−7.6 to +0.7)
- Cost per successful task
- −93.4%
- p95 latency
- 692 vs 887 ms
+−See the analytics pageProof report
When the proof fails, traffic goes back on its own.
The A/B had enough data and failed, so the system sent the call site back to the frontier and recorded why. A button does the same by hand, at any time, for any running test.
- ^The experiment's status, rolled_back, with its reason.
- ^The share served by our model returns to zero for this call site.
- Exposure to the weaker model
- Half of traffic, under 6 min
- Ramp to 95%
- Never happened
- Saved across the run
- 15.7%
+−See the analytics pageExperiments panel
Live traffic outranks an offline score.
Offline scores flatter.
The held-back messages came from the same pool as training, so they over-rated the model. Shadow answers are now scored against the team's corrections, and that score gates the A/B.
A zero threshold is no safety net.
The offline win set the fallback threshold to zero, so hard cases never went to the frontier. The threshold is now set from live, corrected traffic.
The corrections are the asset.
Every message the team re-routed is a labelled hard case, including the ones our model got wrong. The next model trains on exactly those, starting with the intents where it trailed.
- Messages. 2,084 real customer messages from Banking77 (PolyAI), across 12 intents that come in look-alike pairs.
- Frontier. Simulated: right 88% of the time and, when wrong, picks the look-alike intent. Priced at $3 / $15 per million tokens.
- Our model. SmolLM2-135M-Instruct with a LoRA adapter, trained and served on an 8-core Apple M2 CPU.
- Traffic. 1,122 messages before training; 962 never-seen messages for the A/B, at 3 per second.
Run the same loop on your traffic.
It starts with a free spend scan: read-only billing access and a repo scan, no code change, about a day.
Get the free spend scan