GPU vs RAM: KV-Offload Is a Speed Tax, Not a Bigger Brain

Note (this is Phase 1). The numbers below come from the KV-location A/B test (Tier O, Phase 1). They're solid for the question this post asks, which is where RAM offload helps and where it just taxes you. The MoE expert-offload phase (below) is still to come.

Here's the finding up front. Spilling the KV cache into system RAM almost never makes a model smarter. It just makes the same answer arrive slower. There's exactly one case where it buys you something real, and that's one model in five.

The RTX 3080 in this box has 10GB of VRAM. That's not a lot. Modern models want to hold a KV cache, which is the running memory of everything in the context window, and that cache grows with every token you feed it. On a small GPU you hit the wall fast: the weights fit, but there's no room left for a long document's worth of cache.

There's an obvious-looking escape hatch. LM Studio (and llama.cpp underneath it) will happily put the KV cache in system RAM instead of VRAM. I have 32GB of RAM and only 10GB of VRAM, so on paper that's a huge upgrade. Spill the cache into RAM, run gigantic contexts, problem solved.

So I A/B-tested it: same model, same quant, same task, KV-on-GPU vs KV-in-RAM, swept across context lengths from 8K up to whatever each model could reach. The result is the least glamorous finding in the whole benchmark, and the most useful.

Below the VRAM wall, offload is a speed tax. You pay throughput and get nothing back.

The control: identical answers, half the speed

The clean experiment is qwen3.5-9b, which already fits entirely on the GPU. It doesn't need RAM. So if I force its KV cache into RAM anyway, the maths is identical (same weights, same context, same output) and the only thing that can change is speed.

That's exactly what happened. Both arms reached the same 262K context ceiling. But at a working 8K context, decode dropped from 66.8 tok/s on the GPU to 30.7 tok/s in RAM, about 46% of GPU speed. You did the same work, got the same answer, and waited more than twice as long for it.

2× context ceiling, granite (the one real win) · ~46% of GPU throughput, control @ 8K

That's the whole thesis in one model. When the GPU already fits the job, RAM offload is pure overhead. The cache now lives across a slower memory bus, so every decoded token re-reads it more slowly. Same brain, worse reflexes.

The exceptions are where it gets interesting

A flat "offload is bad" would be a boring post. It isn't flat. Placement matters differently per model, because each model hits a different wall first.

granite: the one case where RAM buys you something real

granite-4.1-8b is the model that justifies the whole feature existing. On the GPU it tops out around 65K context. The cache fills the card and it simply won't load anything bigger. Push the KV cache to RAM and it reaches 131K, double the ceiling, while running at about 57% of GPU speed at a working 8K context (48.5 vs 85 tok/s). At the very top end it holds up better still: 43.2 tok/s in RAM at 131K against 56.6 on the GPU at its 65K limit.

That is a genuine capability gain. For granite, KV cache is the binding constraint, so moving it off the card buys you context the GPU could never hold. If your document is longer than 65K tokens, "slower but it actually fits" beats "fast but truncated" every time. A truncated document means the model is blindfolded, not stupid.

nemotron: forcing offload is a blunt instrument

nemotron-nano-12b advertises huge context, and KV-on-GPU already delivers: it reaches 262K natively, running 65+ tok/s at short contexts and degrading gracefully to ~18 tok/s at the very top. Force its KV into RAM and it flatlines at 6.6 tok/s everywhere, about a tenth of its GPU speed, and gains zero extra ceiling. You took a model that was already fine and kneecapped it. There's no trade here, just damage.

the MoE: neutral, because it's the wrong knob entirely

The 35B mixture-of-experts (qwen3.6-35b-a3b) barely cares about KV placement. Both arms sit around 9–10 tok/s. That's the tell. This model is slow because of compute and weights, not cache. Moving the KV cache around does nothing, because the KV cache was never its bottleneck. Its real offload story is expert-weight offload, a different mechanism entirely, and one the simple REST KV switch can't touch.

The rule that falls out of this

The pattern is clean once you see it. Offload only matters at the ceiling.

THE RULE. If your context fits on the GPU, keep the KV cache on the GPU. Offloading it just makes the same answer arrive slower.

If your context doesn't fit, like granite past 65K, then RAM is the cheap way to buy capacity, and you pay for it in speed. That's the only time it's the right call.

The mistake is treating RAM offload as a free upgrade: "I have 32GB, so I have a 32GB GPU now." You don't. You have a 10GB GPU and a slow annex, and the annex is worth using only when the alternative is not running at all.

Knowing exactly when that switch helps and when it taxes you is part of the larger game this series is playing: speccing the cheapest box that still makes an existing person faster, without paying for capacity or throughput you'll never use.

The 2026 hardware angle

There's a tempting economic story underneath this, and it's almost a great deal. System RAM is far cheaper per gigabyte than VRAM. Adding 32GB of DDR costs a fraction of stepping up to a used 3090 (24GB), a 4090, or a 5070-class card. So the offload route looks like buying GPU capacity at RAM prices.

The catch is everything above. You're not buying VRAM cheaply, you're buying capacity cheaply and paying the difference in throughput. For granite at 131K that trade is worth it. For a model that already fits, you've bought nothing and slowed down. Speed is the price of the cheap RAM route, and you should only pay it when you're actually at the wall.

The through-line

This series keeps landing on the same uncomfortable point: paying more buys a different thing than you assume. Part 1's version was about quality. This one is about capacity. Spending RAM doesn't make a model smarter, more faithful, or better at your task, because every arm in the control produced identical output. What RAM buys, at exactly one model in five here, is the ability to hold a document the GPU couldn't. That's it. Worth knowing before you reach for the switch and wonder why everything got slow.

Takeaways

• Below the VRAM wall, KV offload is pure speed tax: identical output, roughly half the throughput (66.8 → 30.7 tok/s on the qwen3.5-9b control at 8K).
• Offload only earns its keep at the ceiling. granite-4.1-8b doubles its context (65K → 131K) at ~57% speed. Every model that already fits gains nothing, and nemotron-nano-12b collapses to ~10% of its speed for zero extra room.
• RAM is cheaper than VRAM per GB, but you're buying capacity, not speed. Only pay the toll when the alternative is a truncated document.

Next in the series: the part the KV switch can't reach, which is offloading MoE expert weights, where the 35B's real story lives.