Two posts back I laid out the first wall: the documents where AI would help you most are the ones you can't legally paste into someone else's chatbot. Last post added the second one. Even when you can use the cloud, a document agent burns through your budget faster than anyone warns you, and the bill never stops.
Both walls point the same way. Don't send the data out. Which leaves the question everyone assumes kills the idea: is the local stuff any good, or is "private AI" still a slow, dumb toy?
I had a selfish reason to find out. I wanted to lower my own AI bill without giving up the thing I actually valued, which was having my data on hand whenever I needed it. A lot of that data is the sensitive kind from Part 1: my health export, my medical records, my own business paperwork. So the real question was never "is local good" in the abstract. It was sharper. Which model is actually good enough, and good enough for which task? That is how a weekend experiment turned into benchmarking thirteen of them.
I didn't take anyone's word for it. I built the thing and scored it head-to-head against a frontier cloud model on identical work.
What I built
A completely offline document agent, running on one used graphics card. A second-hand RTX 3080 with 10GB of memory, the kind of card a teenager had in a gaming PC two years ago.
- A local model reads your documents and builds a searchable index, on the box, nothing uploaded.
- You ask a question. It finds the relevant passages and answers from them.
- No internet round-trip, no per-token bill, no data leaving the building.
Then I gave it a 40-document business corpus and 15 ground-truthed questions. Eight were simple lookups. Seven needed reasoning across several documents, the kind where you ask "which auto-renewing contract conflicts with the budget freeze, and why?" I ran the same questions, on the same retrieved evidence, against a mid-tier frontier cloud model as the gold standard.
The result
~93% of frontier quality · ~$0 per analysis · ~$700 used gaming rig
| Found the right info | Answered correctly | Speed | Cost / question | |
|---|---|---|---|---|
| Local (used 10 GB card) | 100% | 14 / 15 (93%) | instant find · ~1–2s answer | $0 |
| Frontier cloud | 100% | 15 / 15 (100%) | network round-trip | per-token |
Two things to take from that table.
Finding your information is already a solved, free, instant problem at home. The local index pulled the right evidence for every single question in about a millisecond. Full recall, near-zero latency, no cost. The part everyone worries about, "can it even find things in my files", turns out to be the easy part, and it's free.
The entire gap to the frontier came down to one arithmetic mistake. The local model matched the cloud on every lookup and on every hard cross-document inference. It caught a contract clause that conflicted with company policy. It caught a renewal that collided with a budget freeze. Its single miss across 15 questions was adding two invoice totals wrong.
The whole frontier advantage came down to the model mis-adding two numbers. That's a calculator problem, not a comprehension one.
A calculator problem has a calculator fix: give the agent a calculator tool. The reasoning was already there. The model just shouldn't have been doing mental arithmetic. So the honest headline is about 93% of frontier quality, near enough to free, and the files never moved. The one weak spot is the most boring, most fixable error there is.
THE POINT. For document search, triage, and Q&A over sensitive files, a model on your own hardware gets you roughly 93% of frontier quality at about $0 a query, with the data never leaving your control.
Two numbers, so let me keep them honest. The 93% is this test, the retrieve-and-answer agent, 14 of 15. The workhorse inside it, a 9B model called qwen3.5-9b, separately hit 95% of the gold standard on a harder real legal-and-medical synthesis benchmark. Different tests, both real, both landing near 95% of the way there for almost nothing. I won't pretend they're one figure. But they tell the same story.
Who this is actually for
Let me put a face on that 93%, because it isn't the CEO's. The person who feels this most is the admin drowning in the filing. And that is no rounding error of the economy. In Australia, Clerical and Administrative Workers are about 10.5% of the whole workforce, 1.63 million people (ABS, Jobs in Australia 2022-23). In Greater Sydney alone that's roughly 340,000, and since Sydney leans hard into finance, professional services and corporate admin, the real number is probably higher.
That's the job this was built for. Contracts, records, claims, invoices. The exact material you can't upload and can't get through fast enough. I know the shape of it, because I've been the person buried in that paperwork.
Look again at what the benchmark says about that person's day. The tool found the right evidence every time, instantly. That is the digging, the part that eats the afternoon. The one thing it got wrong was arithmetic. Everything that needs judgment about what the documents mean, the conflict, the collision, the "so what do we do now", stays the human's call. As it should.
The setup doesn't take the admin's job. It hands them the evidence in a second and leaves the judgment where it belongs.
That's the whole series in a sentence: the cheapest, most private setup makes the people you already have measurably faster at the work only they can do. Same people, unblocked, for the price of a used graphics card.
The part I actually want you to hear: this is a used gaming rig
Here's what makes this practical rather than a lab curiosity. None of it ran on enterprise hardware. The whole benchmark, indexing, searching, answering, ran on a single second-hand RTX 3080. So the setup a small business actually needs looks like this:
- A used gaming PC. A second-hand RTX 3080 (10GB), or a used RTX 3090 (24GB) for more headroom, roughly $300–700 on the second-hand market. Drop it into any reasonable desktop.
- Free software. Point-and-click tools for the models, plus a couple of open-source pieces for the document agent. No licences.
- Free models. Open-weight ones you download once. No subscription.
- An afternoon. A reasonably tech-savvy person sets it up in a few hours and points it at a folder.
Nowhere in that list is there a data centre, a cloud contract, an enterprise GPU, or a per-seat AI subscription.
The math that should change your mind
| Cloud document agent | Used-gaming-rig agent | |
|---|---|---|
| Up-front | $0 | ~$700, once |
| Per year (1 user) | $1,555, forever | ~$0 |
| 10-person firm / year | ~$15,000, forever | ~$0 |
| Your data | leaves the building | never leaves |
The box pays for itself in about five months against one user's cloud bill, then runs essentially free for years, across as many people and questions as you like. That "10-person / ~$15,000" row is the tell. On the cloud, cost scales with people, because you're renting each of them a subscription. On-box, one purchase equips the whole team. You're buying your people a better tool, not a reason to have fewer of them. And it quietly dissolves the privacy wall from Part 1, because the data never goes anywhere.
The whole arc, in one line
Step back across all three posts and the through-line is the same slightly uncomfortable truth: paying more buys a different thing than you assume. The cloud doesn't buy you privacy, it costs you privacy. The expensive frontier model didn't buy meaningfully better document reasoning here. It bought a calculator you could bolt on for free.
The dilemma from Part 1, powerful cloud AI you can't legally use or private local AI that's a toy, was a false choice the whole time. For the job that actually matters to a regulated or sensitive business, the cheap option turned out to be good enough. And "good enough" means something specific here: good enough to make the person already doing the work faster at it, sitting alongside them rather than standing in for them. The real brains of the operation is still the human. Private, capable AI for your most sensitive documents isn't an enterprise privilege any more. It's a weekend purchase.
Where I actually landed
To close the loop I opened at the top: this was never about ripping the cloud out. It's the balance I wanted from the start. The routine, private, heavy work runs on the box for the cost of electricity, which quietly cut my own AI bill. The data I need stays right here, on hand whenever I reach for it. The cloud gets kept for the jobs that genuinely earn it. Cloud when it's worth it, local for everything that doesn't need to leave the building.
The one thing that setup asks of you is knowing which local model to trust with which task, because there's no single winner. The best calculator isn't the best writer. The fastest model is the one most likely to make something up. Right model, right job. That is the whole reason a weekend experiment became a thirteen-model benchmark, and it's what the technical series digs into.
Takeaways
- The line that carries the series: the cheapest, most private setup makes the people you already have measurably faster at the work only they can do. The ~340,000 admin workers in Sydney are exactly who I mean.
- For sensitive-document Q&A, a used gaming rig hit ~93% of frontier quality at ~$0 a query, and the one miss was bad arithmetic, not bad reasoning. Judgment stays with the human.
- Finding the right evidence in your own files is already free, instant, and completely accurate at home. The hard part everyone fears is the part that's already solved.
- The privacy wall and the cost wall fall together. The data never leaves, and the box pays for itself in about five months.
Want the receipts? The technical series walks through the model-by-model results behind every figure here, starting with 13 local models vs frontier.