The Cost Wall: How a Document Agent Torches Your Cloud Budget

In Part 1 I hit the first wall: the documents most worth pointing an AI at are the ones you legally can't paste into someone else's chatbot. But set privacy aside for a second. Suppose the data were uploadable. Could you just pay for the cloud and be done with it?

No. There's a second wall, and it's made of money. People assume cloud AI is cheap because a single chat costs fractions of a cent. Document work is a different animal, because real document collections are enormous in the only unit that gets billed: tokens.

Your files are bigger than a context window, by a lot

I added up my own sensitive corpus. The health export, the clinical and prescription records, and a few hundred business documents I'd genuinely want an agent working over, and the bill I'd personally be paying to run it in the cloud. Here's what that actually weighs:

That last number has a brutal implication. Yes, 2026 frontier models finally ship 1-million-token context windows. Claude Opus 4.8 and Sonnet 4.6, Gemini 2.5 Pro, and since Anthropic dropped the long-context surcharge earlier this year, at standard rates. It's a genuine jump from the old 200K ceiling. It still isn't enough. At ~4.4 million tokens, my corpus is more than 4× the largest window on the market. You physically cannot load your files in and ask a question.

Here's the counter-intuitive part. Even where a 1M window would swallow the job, loading a million tokens on every question is the expensive way to work, not the cheap one. You pay, in full, to re-read the whole pile each time you ask something. That's why real agents don't do it. They chunk the documents, retrieve only the relevant pieces per question, and re-read those. Stuffed context or retrieval, it makes no difference to the meter. Every token is billed, every time.

Run that in the cloud and it stops being a project cost. It becomes a subscription to your own paperwork.

("But prompt caching!" Fair, and worth saying out loud. Caching can knock about 90% off re-reading identical context, so it helps when you're firing lots of questions at the same stable text. It doesn't rescue this case. It does nothing for the privacy wall, the corpus is still 4× too big to cache in a single window, cache entries expire in minutes, and a retrieval agent pulls different passages per question, so the cache keeps missing exactly where it'd need to hit. It narrows the gap. It doesn't close it.)

The meter, in real numbers

So I modelled an agent doing actual work: one initial pass to understand the whole corpus, then roughly 20 questions a day for a month at ~50K tokens each. Input tokens only, since adding output just widens the gap. The three cloud tiers are real 2026 list prices, not guesses. Economy is a Haiku-class model at about $1 per million input tokens, balanced is Sonnet-class at about $3, premium is Opus-class at about $5. With the long-context surcharge gone, that flat rate holds even at the top of the 1M window, so these figures give the cloud its cheapest honest number.

One full analysis (read + chunk-summarize + synthesize ≈ 3× the corpus, ~13M input tokens):

An agent in ongoing use (initial analysis + ~43M tokens/month):

$1,555/yr balanced-frontier, per user · 4× too big for even a 1M window · ~$0 local, per year

And here's the part that turns a line item into a budget problem. It's recurring, and it's per corpus, per user.

• One employee, one document set: about $1,555 a year at balanced rates, every year.
• A ten-person firm, each with their own files: about $15,000 a year, forever.

Read that second figure the right way. It scales because you're equipping ten people, not replacing them. Ten staff each with their own document assistant, billed forever, for the privilege of asking questions about paperwork you already own. Now put the whole thing on-box instead. One ~$700 local box (a used 24GB GPU) pays for itself in about 5 months against a single user's cloud bill, then runs essentially free. For that person, or for all ten of them.

You "blow through usage" two ways, not one

The dollar meter is only half the story. Even if the budget were fine, you slam into operational walls.

The first is the context wall. Even a 1M-token window can't hold a single corpus. It's 4× too big to load at once, so everything has to be chunked and retrieved, which is what multiplies the token count in the first place.

The second is the rate-limit wall. Lower API tiers cap how many tokens you can push per minute. Scan a real corpus and you get throttled. The job crawls or stalls before the invoice even lands.

So the second wall is just as solid as the first. Privacy rules out the cloud for sensitive files. Cost and limits rule it out for large document work even when privacy doesn't. Both point the same direction.

THE POINT. A document agent isn't a few cents a chat. It's a few thousand dollars a head, every year. The through-line of this whole series is that paying more buys a different thing than you assume. With document AI, more money doesn't buy you the answer you can't otherwise get. It buys you a metered, throttled, recurring bill for work an on-box model can do for the cost of electricity.

So the obvious question

If the data never leaves the building, both walls vanish at once. No third-party retention, no rate limits, no per-user subscription to your own files. The only thing standing in the way is the assumption everyone repeats without testing, that local AI is a slow, dumb toy.

Let me be straight about what going local does not do. It does not hand you a bigger window. A 10GB gaming card holds far less context at once than a frontier model. Try to shove a million tokens through it in one pass and you'll wait a long, tedious while. It is emphatically not an "it all fits" machine. But that's fine, because brute-forcing an entire corpus into one context is the wrong move on the cloud too, which is what the whole first half of this post just showed. The right move in both places is retrieval: pull the handful of passages a question actually needs. The local box just does that part for free, on-prem, and without anyone metering it.

That assumption deserves to be measured, not asserted. So I built the offline agent, a fully on-box RAG loop on a used gaming rig, and ran it head-to-head against a frontier cloud model on real legal and medical document queries. The results genuinely surprised me.

Takeaways

• A real document corpus runs to ~4.4M tokens, more than 4× even the 1M-token windows frontier models ship in 2026. So a cloud agent has to chunk, retrieve, and re-read, and every read is billed.
• At balanced-frontier rates that's about $1,555 per user per year, forever, against roughly $0 on a one-time ~$700 local box that pays for itself in about 5 months.
• Cost and rate limits rule out the cloud for large document work even when privacy doesn't. Both walls point the same way: keep the data on-box.

Next: About 93% of Frontier Quality, for ~$0, the offline answer on a used gaming rig.