[Local LLM] Why a 284B fits a 128GB GB10 at long context: DeepSeek-V4-Flash attacks the KV cache, not the parameter count

The short version

If you've read that some giant model "fits in 128GB," the natural question is how — 284 billion parameters should not fit in a 128GB box, and they don't fit comfortably. But the parameter count turns out to be the boring half of the problem. The half that actually decides whether a local model stays usable is the KV cache: the running memory of everything in the conversation so far. Every token you feed in leaves something behind that the model has to keep around and re-read on every future step. At short context it's nothing. At 64K tokens, on a conventional architecture, it's tens of gigabytes and it grows the whole time you talk — and eventually it, not the weights, is what runs you out of memory.

DeepSeek-V4-Flash is built to make that pile small and cheap to read. It does two separate things. First, it never stores a full-size cache: most layers compress each token's KV hard — 4x on half the layers, 128x on the other half — so 64K tokens of cache comes out under a gigabyte. Second, when it thinks, it doesn't re-read the whole pile — a lightweight indexer picks the few hundred most relevant compressed rows and reads only those. The result is that 64K tokens of cache is ~871MiB, and each step touches a few hundred rows instead of tens of thousands. That's the whole reason a 284B is pleasant at long context on hardware that, on paper, has no business running it.

---

The setup

Parts 1 through 4 of this series were operational: get the 284B running on one GB10, pick the daily settings that make it bearable, argue that Q2 weights still beat the small model that fits, and chase down a KV-cache eviction storm a month later. This one is the architecture piece the rest lean on — the "why does any of that work" underneath the ops.

The thesis is one sentence: at long context, the enemy of a local LLM is not the parameter count, it's the KV cache footprint — and DeepSeek-V4-Flash is architected specifically to attack the KV. Everything below I checked against ds4's implementation of the model and DeepSeek's V4 architecture paper, whose §2.3 ("Hybrid Attention with CSA and HCA") is where the attention design lives.

One number to anchor on: DeepSeek-V4-Flash is 284B total parameters, ~13B active per token — an MoE with 256 routed experts, top-6 plus one shared, 43 layers, hidden size 4096. So the model is already sparse on the compute side: 13B of 284B does the work each step. What this post is about is the second, sneakier kind of sparsity — sparsity on the memory side.

Hybrid attention: SWA + CSA + HCA compress the KV to ~871MiB at 64K

The first lever is that V4-Flash never stores a normal KV cache. A conventional model keeps a full key and value vector per head per token — that's the thing that makes the cache enormous. V4-Flash instead runs three different attention layer types, staggered through the stack, and — past the first two — every layer stores a compressed cache. In ds4's code the compressed entry is attn_comp_kv, 512-wide, and the per-layer compression schedule is read from the config field deepseek4.attention.compress_ratios rather than hardcoded:

• Layers 0-1: a plain sliding window (SWA). The first two layers don't compress; they just cap how far back they look. Cheap, local, and they never grow.
• Even layers 2-42: CSA, compression ratio 4. Compressed Sparse Attention squeezes every 4 tokens into one compressed entry, then (below) reads those entries sparsely.
• Odd layers 3-41: HCA, compression ratio 128. Heavily Compressed Attention squeezes 128 tokens into one entry — a brutal ratio — and reads them densely. At 64K that's only ~512 entries for the whole layer.

So it's not one knob, it's a schedule: half the stack pays a 4x compression and reads sparsely, half pays a 128x compression and reads what little is left. The config decides the ratios; you don't tune them, and you shouldn't.

The payoff is a KV cache that is almost embarrassingly small. Reading through ds4's cache accounting, 64K tokens comes out to about 871MiB (that's the logical/CPU figure — the CUDA graph reserves a bit more, ~1.2GiB, for its persistent prefill cache). On a conventional full-attention layout the same context would be tens of GB. This is the single fact that reframes the whole "how does a 284B fit" question: at the context lengths I actually run, the KV cache is a rounding error next to the ~80.8GiB of weights, not the other way around.

The lightning indexer (DSA): reading only the top-512 compressed rows, ~3% per step

Compression shrinks how much you store. The lightning indexer shrinks how much you read on every decode step — a different cost, and the one I find most interesting.

On the CSA layers, V4-Flash does top-k sparse attention: a lightweight indexer scores the compressed entries and keeps only the best ones. This is the selection half of DeepSeek Sparse Attention (DSA). In ds4 the parameter is n_indexer_top_k = 512 (the Pro variant uses 1024), and the indexer has its own small 128-dim keys — index_comp_kv. On CPU it produces a boolean mask (comp_allowed); on GPU it passes the selected row IDs straight into an indexed attention kernel.

The arithmetic is the point, and it's where the draft I started from was wrong. The indexer selects among the compressed CSA entries, not raw tokens. At 64K, a CSA layer has already folded the context 4x into ~16K compressed entries — and top-512 is about 3% of those. Add the 128-token sliding window and a CSA layer reads roughly a few hundred rows per step, out of 64K tokens of raw context. The HCA layers don't even need a top-k: compressed 128x, they only have ~512 entries in the first place. Either way, decode reads a few hundred compressed rows per step, not the whole context — and it gets more effective the longer the conversation gets, because the top-k budget is fixed while the context grows. The thing that normally makes long context painful — more tokens, more to re-read — is exactly what this is designed to blunt.

The payoff: 284B fits 128GB, and there's nothing left to quantize in the KV

Put compression and sparse reads together and the "how does a 284B stay usable at long context on a 128GB GB10" question dissolves. The weights are the fixed cost (~80.8GiB at Q2). The KV cache — which on a conventional architecture would grow without bound until it runs you out of memory — is instead ~871MiB at 64K, read a few hundred rows at a time. The box has room, and decode isn't dragging a full-context KV read behind every token.

This also settles something I spent real time on in Part 2: you cannot quantize the KV cache to save memory here, and there's nothing to gain by trying. Two reasons, and they reinforce each other. First, the compressed KV is already stored at reduced precision — the paper keeps BF16 on the RoPE dimensions and FP8 (E4M3, in 64-wide chunks with a power-of-two scale) on the rest, so the non-RoPE part is already 8-bit, not f16 waiting to be crushed. Second, ds4's CUDA attention path doesn't even support an f16 compressed-KV cache — it bails out when that mode is set, so the path simply isn't there. And even if it were, the prize would be a fraction of 871MiB. The architecture already did the memory work upstream, so the usual "quantize the KV to fit more context" move is both blocked and pointless. (This is the mirror image of the weights story in Part 3: crush the weights hard, leave the KV alone.)

The indexer already tracks dense attention at 93-96% — which is why bolting on more doesn't help

One last finding, because it foreshadows where this series goes next. The obvious question about top-512 sparse attention is: what does it cost you? If you're only reading a few hundred rows, are you missing the ones that mattered?

In my own measurements — the indexer's selection versus what a true dense-attention pass would have weighted — the answer is: barely. The native indexer overlaps with true dense attention by 93-96% across the layers I checked; it is picking almost exactly the positions a full-attention pass would have, at a fraction of the reads. (That's my measurement on this model, not a figure from the paper — I'll put the full numbers up in the next part.) That near-optimality has a practical corollary that cost me an experiment or two to learn: bolting an external sparse-KV scheme on top of V4-Flash doesn't help, because the model already ships a very good one. The headroom you'd be reaching for is the 4-7% the native indexer leaves on the table, and an external heuristic isn't going to reliably beat a trained-in indexer for that sliver.

Takeaways

Where the time went

It wasn't in reading the model — once you have the right names, SWA, CSA, HCA and the indexer are each a clean idea. It went into un-learning the instinct that the parameter count is the memory problem. I started out thinking about the weights: budgeting VRAM for ~80GiB and treating context as a second-order worry. The architecture inverts that. The weights are a fixed slab you pay once, and the thing that would have grown unbounded — the KV — is the part DeepSeek engineered down to a rounding error. Every hour I spent trying to "make room" by squeezing the KV was an hour spent on the half of the problem that was already solved.

Reusable diagnostics

When you're sizing a long-context model on fixed hardware, compute two numbers separately: the weight footprint (fixed, quantizable) and the KV footprint at your real context length (grows with tokens, architecture-dependent). For a conventional attention model the second number is the one that bites; for a compressed + top-k model like V4-Flash it's the one that's already been defused. And a specific tell: if a model's KV cache is already stored FP8 and its runtime won't even run an f16 KV path, that's the architecture telling you the KV work is done upstream — stop hunting for a quantization win there and spend the effort on prefill reuse instead (which is Part 4).

The general principle

At long context, the footprint that decides whether a local model is usable is the KV cache, not the parameter count. A model architected to attack the KV — compress what you store, then read only the fraction that matters — will fit and stay fast on hardware that its parameter count says it has no right to run on. Read the KV path before you read the weight count.

Conclusion

The short version of the short version:

1. A 284B fits a 128GB GB10 at long context because the KV cache is tiny, not because the weights are. ~80.8GiB of Q2 weights, ~871MiB of KV at 64K.
2. Hybrid attention compresses what you store: a sliding window on layers 0-1, CSA (4x) on the even layers, HCA (128x) on the odd — a schedule baked into the config.
3. The lightning indexer (DSA) compresses what you read: top-512 of the compressed CSA entries per step — a few hundred rows, ~3% at 64K, and more effective the longer the context.
4. Don't try to quantize the KV. It's already FP8 on the non-RoPE dims, already small, and ds4's CUDA path won't run an f16 KV cache. The memory win was architected in; go reuse prefill instead.

The parameter count is what everyone quotes. The KV cache is what actually decides whether the thing is usable a month into daily use. V4-Flash is built around that distinction, and once you see it, the rest of this series reads differently.

---

Also in this series: Part 1 — My first Q2 model looked broken; the real culprit was a parser that couldn't read DSML · Part 2 — Running a 15 tok/s 284B as your daily agent brain: the settings that make it bearable · Part 3 — Weights win: a 284B crushed to 2-bit still beats the small model that fits · Part 4 — My 284B agent quietly stopped reusing its KV cache: the ds4 evict storm · Part 5 — Depth-1 MTP is a workload-dependent trade, not a global speedup · DeepSeek-V4 architecture paper (arXiv 2606.19348)