This is the receipts post. The business series argues you can keep sensitive data on-box and still do real work with it. This is where I show the actual numbers behind that claim. It's also the answer to the question that drove me here. When I set out to cut my own AI bill without giving up my data, I needed to know which local model to trust with which job, because, as you'll see, there's no single winner. I put 13 local models on one RTX 3080 (10GB VRAM) and scored every one against frontier "gold" answers. Economy-frontier and balanced-frontier cloud tiers set the 100% quality ceiling, with a stronger model acting as judge for the open-ended stuff. Where there's objective ground truth (code that executes, arithmetic with a known answer, dates that parse), I auto-scored. Where quality is subjective (legal synthesis, summary prose), a judge scored each local answer as a percentage of gold.
One caveat, in the spirit of measuring honestly. The benchmark is still being finalised. Two reasoning-only models are being re-run, and five are missing their real-document scores, so treat any single figure as current-best rather than final. But the shape of the result is already clear, and it's more interesting than "local good" or "local bad."
95% of gold on real legal/medical docs · 9B params, runs on 10GB · $0.0004 per 1M output tokens (electricity)
The leaderboard
Composite is a rough mean of the objective and judged percentage columns: codegen, the hard-coding tier, faithfulness, health-compute, health-dates, and health-synthesis. It's a blunt instrument on purpose. The interesting stories are in the per-task splits, not the average. Top rows, trimmed to the columns that matter:
| # | Model | Params | tok/s | Composite | Stand-out | Weak spot |
|---|---|---|---|---|---|---|
| 1 | qwen3.6-35b-a3b (MoE) | 35B-A3B | 12.9 | 93 | faithful + best synthesis (88); 100% code & hard-code | slow |
| 2 | qwen3.5-9b | 9B | 57.2 | 89 | 95% of gold on real docs; faithful | health-compute 50% |
| 3 | gemma-4-12b | 12B | 40.7 | 88 | 100% code; perfect dates | health-compute 50% |
| 4 | gemma-4-e4b | 7.5B | 109.9 | 88 | 100% health-compute, fast | math 50% |
| 5 | gemma-4-e2b | 4.6B | 159.5 | 87 | 90% health-compute, fastest accurate | weaker synthesis (62) |
| 6 | gpt-oss-20b | 20B | 19.1 | 77 | 100% math + code | faithfulness ~67% |
| 11 | mistral-7b-v0.3 | 7B | 119.8 | 61 | fast; 100% code | fabricates (synth 25) |
| 12 | lfm2.5-1.2b | 1.2B | 334.6 | 43 | fastest of all | unsafe; hallucinates |
Two models I've left off the visible cut (deepseek-r1-8b and glm-4.6v-flash) scored badly (composites 39 and 64), but with an asterisk. They're reasoning-only, and the first run gave them too small an output-token budget, so their "thinking" truncated the actual answer into a false zero. Those two are being re-run. I'm flagging it rather than burying it, because the honest version of this leaderboard has provisional rows in it. The corollary, that "reasoning" models aren't reliably safer, is a claim I'm holding as provisional until those re-runs land, precisely because that budget bug is exactly the kind of thing that could be manufacturing the effect.
Stability is a gating criterion, not a footnote
Before quality even enters the conversation, a model has to finish the job on the hardware you actually have. On this 10GB card, the dense reasoners aren't merely slow. They're operationally unstable, and disqualified. deepseek-r1 dropped the connection mid-tier and ran 2–3.5 hours per model. ministral-3-14b did the same at 13 tok/s. granite-4.1-30b at 5 tok/s couldn't finish a single health generation before Node's timeout killed it. An unstable model is not fit for purpose regardless of how it might score. It doesn't get caveated, it gets cut. The models that survive this gate (qwen3.5-9b, the MoE models) are the ones that run reliably and fast on cheap hardware, and that's not a coincidence. Reliability on a 10GB box is the first thing the leaderboard is actually measuring.
Explicit-evidence retrieval is easy. Inference over an unstated conflict is hard.
This is the single line that runs through every tier. The real-document tier uses actual workers-comp and medical claim documents, full of PII, kept in a gitignored folder that never touches a transcript. I asked four questions of escalating difficulty. The two easy ones ("what does the notice say?") are near-gold on basically every model. The gap opens on the hard two. q3 is an unstated conflict between two documents. q4 asks whether a suspected hereditary condition was ever genetically confirmed, where the answer ("no, never") is buried in a record showing the confirmatory test was cancelled. You have to reason about the absence of evidence.
Explicit-evidence retrieval is a solved, free, on-box problem. Inference over an unstated conflict is where local still separates from frontier.
qwen3.5-9b handled both hard queries and landed at 95% of gold across the four. That's frontier-comparable on a 10GB box. By contrast, deepseek-r1 flipped the q3 conflict (concluded there was none), and the speed-king lfm2.5 rambled and invented. Same documents, same retrieval, wildly different inference.
Computation ≠ communication
Here's the finding that breaks "just pick the top model." The best calculators are not the best writers. The health-data-science tier is an Apple Health export of ~59,000 data points, roughly 2.5 million tokens, about 20× any model's context, so it's a data-science test, not a reading test. On it, the small Gemmas own the arithmetic. gemma-4-e4b scored 100% on computation, e2b scored 90%.
But flip to the synthesis sub-task (write a faithful 6-bullet health summary, judged, penalised for inventing a metric) and the ranking inverts. The 35B MoE wins at 88. nemotron-nano-12b lands an 84 despite being mediocre at the actual math. Compute-champ gemma-e2b drops to 62. Different job, different winner. You don't buy one model, you route to the one that fits the task.
Bigger isn't reliably better
The cleanest counter-example is two siblings. gemma-4-e4b is 7.5B. gemma-4-12b is, well, 12B. On health-compute the smaller one scores 100% and the bigger one scores 50%. The 4.6B e2b also beats the 12B there (90% vs 50%). Meanwhile nemotron-12b, a perfectly respectable 12B, collapses on tasks that 8B models ace: 33% on codegen, 17% on the computational tier, where qwen3.5-9b and the Gemmas sit at or near 100%. Parameter count is a budget line, not a quality guarantee.
Speed correlates with risk
The fast tier is the dangerous tier. lfm2.5 runs at 334 tok/s and is the fastest thing I tested by a mile, and it genuinely hallucinates on real-document synthesis (synth score 30, faithfulness 33%). mistral-7b does 120 tok/s, aces code, and then fabricates under writing pressure (synth 25). The faithful models are slower: qwen3.5-9b at 57 tok/s, the 35B at 13 tok/s. The "free and instant" local dream has a measurable faithfulness cost, and it shows up exactly in the 100–334 tok/s band.
RAG QA needs a judge, not string-matching
A credibility note, because it nearly fooled me. I built a fully on-box document-search agent, a local embedding index plus qwen3.5-9b as the answerer, over a 40-document business corpus with 15 ground-truthed queries. Retrieval was a non-event: 100% recall, sub-millisecond, $0. Finding the evidence is solved.
Note. My first scorer was a naive keyword matcher, and it reported the local agent at 67%. That number was wrong. The matcher was failing on "seven" ≠ "7" and prose dates ≠ ISO format, counting correct answers as misses because the bytes didn't line up. Swapping in a semantic judge gave the true score: local 14/15 (93%) vs gold 15/15 (100%) on identical retrieval. The entire frontier advantage was a single cross-document arithmetic sum the local model mis-added, and that's tool-fixable. Give the agent a calculator. If you're benchmarking RAG with string equality, you're under-reporting your own system.
What this actually means
The through-line: paying more buys a different thing than you assume. It does not reliably buy you better explicit retrieval. That's free and on-box. It does not buy you better arithmetic than a 4.6B Gemma. What the frontier still wins is the hardest inference and the best summary prose, and even there a 9B lands at 95% of gold on the real-document synthesis tier.
Two "9B-at-frontier" numbers run through this series and they are not the same test, so keep them straight. The 95% is qwen3.5-9b on the real-document legal/medical synthesis tier (judged as a percentage of gold). The 93% is the separate built RAG agent, the on-box retrieval-plus-answer stack scoring 14/15 against gold's 15/15. Same model, two different measurements. I cite both because conflating them is how benchmarks start lying.
On cost: local marginal cost is roughly $0.0004 per 1M output tokens (electricity) against frontier rates of single-digit-to-low-tens of dollars per 1M. For high-volume, privacy-sensitive document work, a 9B model at 95% of gold turns a recurring per-token cloud bill into a fixed hardware cost. That's the whole argument, and the data backs it. And that's the point of running this leaderboard at all. It's what tells you a cheap local model is good enough to make the person already doing the work measurably faster. It sits alongside them and hands them the evidence. It doesn't stand in for them.
If you want one pick for sensitive document analysis on a 10GB box, it's qwen3.5-9b. If you want throughput for bounded grunt work, gemma-4-e2b is absurd accuracy-per-watt. And if you want a single model that's best at everything, that one isn't on the list, and the point of this post is that it isn't on anyone's list.
Takeaways
- Paying frontier prices buys the hardest inference and the best prose, not better retrieval or arithmetic, both of which run free and on-box.
- Don't pick one model, route per task. A 4.6B Gemma out-computes a 12B, and speed above ~100 tok/s comes with a measurable faithfulness cost.
- For privacy-sensitive document work, qwen3.5-9b hits 95% of gold on a 10GB GPU at ~$0.0004 per 1M output tokens. That's a fixed hardware cost instead of a recurring cloud bill.
Next: GPU vs RAM offload.