[Junk-Tier Big Models #3] A 284B MoE on ONE 2080 Ti — and it beats my DGX Spark

In plain English: why a 284B model fits on an eight-year-old card

Here's the version for a friend who doesn't do this for a living.

"284 billion parameters" sounds like it needs a server rack. It doesn't, because this model is a Mixture-of-Experts — a huge committee where only a handful of members speak per word. DeepSeek-V4-Flash keeps 256 specialists in every layer and wakes exactly 6 of them for any given token. The other 250 are dead weight at that instant.

Dead weight doesn't need expensive graphics memory. You can park it in ordinary system RAM — which on a junk-tier server is cheap and plentiful — and pay the cost only when a specialist actually gets called. That's expert offload: the busy parts stay on the GPU, the napping parts live in RAM.

What surprised me is how little the GPU ended up mattering. I expected a clean story where more cards meant more speed. Instead one card, two cards and three cards all landed within about 4% of each other, and the whole thing still outran a purpose-built 2025 AI machine sitting on the same network.

---

The rig, the model, and what I was actually asking

The lab is the same junk-tier box from Part 1: an EPYC 7402 (24 cores / 48 threads), three modded RTX 2080 Ti 22G cards, and 128GB of eight-channel DDR4. Compute capability 7.5 — Turing, launched 2018.

The model is Huihui's abliterated Q2 conversion of DeepSeek-V4-Flash: 86,720,111,200 bytes on disk, 284,334,567,511 parameters, IQ2_XXS quantization — 2.0625 bits per weight for that type, though the mixed file averages about 2.44 bits per parameter. Architecture deepseek4, 43 blocks, 256 routed experts per block, 6 active per token.

The question was not "can it load." It was whether I could move DeepSeek-V4-Flash off my DGX Spark — where it had been the resident model for weeks — onto the junk rig, and free the Spark up for training. That only makes sense if the junk rig is at least as fast.

One build note that saved the whole attempt: current upstream llama.cpp compiles and runs deepseek4 on sm_75. I used main at 0e4a0362239713ea95a6864a17a8de4b0ad90d62 with CUDA 12.4 and CMAKE_CUDA_ARCHITECTURES=75. Older builds on this box didn't have the DeepSeek-V4 implementation at all, which is a very different failure from "Turing can't do it."

The first benchmark said 14 tok/s and "the third card is useless" — both were my own config

The first numbers came back discouraging: 14.22 tok/s on two cards, 13.96 on three. The third card made decode 1.8% worse. Read literally, that says GPUs don't help this workload and the junk rig loses to the Spark.

Then I looked at what the run actually used. Peak VRAM was 5.0 and 4.4 GiB — on cards with 22 GiB each. Two thirds of the GPU memory in that box never got touched.

The cause was one flag. -ncmoe N moves the routed expert tensors of the first N blocks onto the CPU, and the benchmark had been run at -ncmoe 43 — every single block. All 43. With every expert on the CPU, the third GPU had nothing left to accelerate.

That is not a hardware finding. It's a measurement that answered a question nobody asked, and it nearly ended the project on the first day. The tell was in the report the whole time — peak VRAM 5 GiB out of 22 is not a hardware ceiling, it's an empty room.

Fill the VRAM and it's 18.6 — but one card already gets 17.85

So I swept -ncmoe downward, pushing experts back onto the GPUs until they wouldn't fit. Three cards, -n 256, three repetitions per point, median reported:

CPU experts	Peak VRAM (MiB)	tg256 tok/s
43 blocks (100%)	9,896	13.9125
30 blocks (69.8%)	31,914	16.4087
20 blocks (46.5%)	49,164	17.4036
13 blocks (30.2%)	61,264	18.6241

The row that matters is the last one: 61,264 MiB is 92.8% of the 66,000 MiB llama.cpp reports across three cards. That's as full as this rig gets, and it buys 18.62 tok/s — 33.9% over the all-CPU baseline.

Then I ran one card, on its own, with -ncmoe 38: only 5 blocks of experts on the GPU, 38 on the CPU. It measured 17.85 ± 0.21 tok/s.

One card. Ninety-eight percent of two cards, ninety-six percent of the fully-loaded three-card configuration, while leaving 38 of 43 expert blocks on the CPU.

Before the run I'd said one card would land around 20. The AI assistant running the benchmark with me predicted 14–16. Neither of us was right, but 17.85 is a lot closer to my guess than to its — and I wasn't being clever. I just didn't believe the card count was the thing that mattered.

That comparison is uneven in one way: the single-card figure is a 128-token generation and the multi-card rows are 256-token generations, so the single card gets a marginally easier test — decode drifts down slightly as the KV cache grows. The conclusion still holds. Serving the same single-card config today, the server reports 17.47 and 18.03 tok/s on back-to-back 128-token completions, which brackets the benchmark number rather than contradicting it.

That's the soft floor. Decode lands near 18 tok/s almost regardless of how much of the model you get onto GPUs.

The empirical argument is the strong one: one card and three cards land 4% apart, even though the three-card run has 30 more blocks of experts sitting in fast memory. Whatever is binding here, it isn't where the experts live.

The arithmetic turned out more interesting than I expected, and it took a fact-check to get right. About 13B parameters fire per token — but you can't price them at the file's 2.44-bit average, because this GGUF is quantized asymmetrically. The 277B routed-expert parameters are the Q2 part, and only 6 of 256 fire; the always-on parameters — attention, shared experts, output — are mostly Q8 or F16. Parsing the tensor directory and taking the top-6-of-256 share: routed experts cost 1.83 GB per token, the always-on remainder costs 7.73 GB. About 9.56 GB per token, an effective 5.76 bits per active parameter — more than double what the file average implies.

That reprices both ceilings. Eight-channel DDR4-2133 at 136.5 GB/s gives a RAM-only ceiling near 14.3 tok/s; a 2080 Ti's 616 GB/s gives an all-VRAM ceiling near 64.4. Measured 17.85 sits above the pure-RAM figure — which is exactly what hybrid placement is supposed to buy — and 3.6x below the all-VRAM one.

So bandwidth is doing real work here. What it doesn't explain is the 4%. Moving those 30 blocks shifts real bytes from the slow pool to the fast one; it should have closed part of that 3.6x gap. It didn't.

That unclosed gap is the soft floor — per-token latency that no amount of memory placement pays down. Scheduling, kernel launches, the CPU/GPU round trip, expert selection.

The third card costs 8.5%, and only a fixed-split A/B shows it

There's a sharper result hiding in the sweep, and it only appears if you hold the CPU/GPU split constant and change nothing but the number of GPUs:

GPUs	CPU experts	tg256 tok/s
2	23 blocks	18.2618
3	23 blocks	16.7048

Same model, same quantization, same 53.5% of experts on the CPU. The third card makes it 8.5% slower. Each extra device boundary costs real time, and in the naive sweep that cost hides inside the capacity gains — more cards let you move more experts onto GPUs, so the line still goes up, and you never see what the extra card charged you.

This is the number that killed a hardware purchase. A capacity-only extrapolation of the three-card curve fits speed = 20.4434 − 0.14898 × N with R² = 0.9770 — six sweep points, four of which are in the table above — which projects about 20.4 tok/s if every expert were GPU-resident — roughly four cards' worth of VRAM. Apply the measured per-card penalty and the same four-card configuration lands back around 18.6–19.4. Buying a fourth card to chase 20 tok/s would have been paying for a projection that a control I already ran contradicts.

Two gotchas that cost hours: the split default, and load-is-not-run

Two failures in this sweep look like "out of memory" and are not.

The tensor split default piles experts onto the last card. llama.cpp's default layer split divides by block count, so GPU-resident experts — which live in the back half of the model — end up concentrated on the final GPU. At -ncmoe 30 that asked CUDA2 for a single 25,091.43 MiB allocation on a 22GB card and died, even though the three cards together had plenty of room. Aggregate capacity means nothing if one device is asked for more than it has. Every sub-43 point needs an explicit -sm layer -ts <split>.

A model that loads can still abort on the first token. At -ncmoe 13 the split 22/10/11 loaded cleanly, then hit a CUDA abort during the first prompt with GPU0 at 21,984 of 22,000 MiB. Rebalancing to 21/10/12 — the same 61,264 MiB total — runs fine. The warmup and prompt graph need workspace that the loader never accounted for, so any capacity scan has to run real computation before it records a point.

Two MTP dialects, and neither one runs

With decode pinned at the soft floor, the only remaining lever was speculative decoding — where a small fast model drafts several tokens and the big model verifies them all in one pass, so a good guess buys several tokens for one expensive step. DeepSeek-V4-Flash ships MTP (multi-token prediction) weights for exactly this.

The first attempt failed at the door:

general.architecture = deepseek4_mtp_support
error loading model: unknown model architecture: 'deepseek4_mtp_support'
failed to load draft model

The main model had loaded first, cleanly — 44 of 44 layers offloaded, 20,627.70 MiB on CUDA0 and 21,315.96 MiB on CUDA1. Only the draft was rejected. So there was no acceptance rate to report and no speculative decode to measure; sweeping the draft length would have re-run the same loader gate three times.

I read that as "MTP is a dead end on this hardware" and wrote it down. That was wrong twice over.

Wrong the first time: the model card points at a dedicated fork, Fringe210/llama.cpp-deepseek-v4-flash-cuda, and I hadn't used it. So I built it — and it compiled on sm_75 without complaint. (Its first load then hung, which turned out to be mmap; --no-mmap fixes it.) The fork rejects deepseek4_mtp_support too.

Wrong the second time: I'd assumed the two MTP files were the same thing under different names, and that editing an architecture string would get me past the loader. Comparing tensor names showed they really are different: mainline expects blk.N.nextn.* with eh_proj, embed_tokens, shared_head; the antirez lineage uses mtp.0.* with attn_q, attn_kv, attn_sinks.

Then I over-corrected. I wrote these down as two incompatible architectures, which is also wrong. Both do a projection, then a full attention block, then MoE — what differs is naming, tensor layout, and how the projection is packed (mainline concatenates the two inputs into one matrix; the antirez side keeps them separate, which is algebraically the same thing). That needs a real converter, not a rename. But it is a packaging difference, not two different designs.

Checking which dialect a given file speaks doesn't require downloading it. GGUF puts its metadata at the front, so an HTTP range request over the first few megabytes is enough to read the tensor names out of an 86GB file — parsed with plain Python, since this box has no numpy.

That's how I found that ddh0/DeepSeek-V4-Flash-GGUF publishes a standalone MTP file in mainline's format: blk.43.nextn.*, architecture deepseek4, nextn_predict_layers 1. So a file in the naming mainline expects does exist. That turned out to matter less than I thought.

The real blocker: DeepSeek4 MTP isn't implemented, anywhere in llama.cpp

It still wouldn't have worked, because mainline hasn't implemented any of this.

src/models/deepseek4.cpp never reads nextn_predict_layers, never loads nextn tensors, and builds no MTP graph. The support isn't half-present or misconfigured — it isn't written. Architectures that do support multi-token prediction read that hyperparameter and build a separate draft graph; deepseek4 does neither. The fork carries a TODO in the same place.

That also means the ddh0 file wouldn't have loaded, for a reason I found by reading the loader rather than running it: mainline expects a complete 44-layer model and throws on the first missing tensor, and a standalone blk.43-only file has none of blk.0 through blk.42. Creating an MTP context separately warns and returns null when the layer count is zero.

I read the source; I did not run that file. The source reading made running it moot — but "the naming is right" and "it would have worked" are different claims, and only the first one is mine.

So the honest ordering of walls, after two wrong answers:

1. Not a hardware wall — Turing compiles and runs the main model fine.
2. Not a build wall — both mainline and the dedicated fork build on sm_75.
3. Not a naming wall — a file in mainline's own nextn naming exists.
4. A runtime wall — DeepSeek4 MTP is not implemented in llama.cpp at all.

The other engine, antirez/ds4 (DwarfStar), does support this properly — it's the stack my DGX Spark runs. But it's an independent MIT project with its own model format, not a llama.cpp variant, and its README's oldest tested hardware class is Ada Lovelace, describing those cards as "old-ish." Turing is four years older than that. Porting it is a project, not a config change.

Worth stating plainly, because I nearly shipped it wrong in an earlier article and the fact-check caught it: antirez/ds4 is antirez's own project, not DeepSeek's official engine.

What shipped: one card, 262K context, 16GB of VRAM

The deployment follows the soft floor rather than fighting it. DeepSeek-V4-Flash now runs on one 2080 Ti, leaving the other two cards for other work:

llama-server \
  -m Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2.gguf \
  -ngl 99 -ncmoe 39 -c 262144 \
  -fa on -ctk q4_0 -ctv q4_0 \
  -t 32 -tb 32 -b 2048 -ub 512 \
  --host 0.0.0.0 --port 8093

That holds a 262,144-token context in 16GB of VRAM, with about 67GB of system RAM for the sleeping experts. The context is nearly free because of how this model does attention: one KV head, MLA-style compression, a 128-token sliding window, and alternating compression ratios from 4 to 128. Working through llama.cpp's DeepSeek4 cache layout with -ctk q4_0, the persistent KV cache at 262K comes out around 480 MiB — hundreds of megabytes, not gigabytes. The model advertises a 1M-token window; 262K is where I stopped, because past that the numbers stop meaning anything for how I use it.

-ncmoe 40 leaves three blocks on the GPU and fits; -ncmoe 35 wants eight and OOMs. The window is narrow and it is worth finding.

One 2080 Ti: 17.4 tok/s, against the DGX Spark's 14.75

Here is the comparison the whole exercise was for. Same script, same prompt, 128 tokens, temperature 0, three runs each, both boxes measured within a minute of each other:

Box	Engine	Warm end-to-end tok/s
ai-lab, one 2080 Ti 22G	llama.cpp, -ncmoe 39	17.41 / 17.36
GX10 DGX Spark (GB10)	antirez ds4	14.75 / 14.69

One roughly NT$11,000 used card, modded from 11GB to 22GB, running a 284B model about 18% faster than the purpose-built 2025 machine that had been serving it.

Two caveats, because this number is easy to over-read. The two sides run different engines and different quantizations — llama.cpp with an abliterated IQ2_XXS GGUF versus antirez ds4 with its own format — so this compares two real deployments as they actually run, not two pieces of silicon under controlled conditions. And it is one workload: short prompt, 128 tokens, single stream.

There's a smaller lesson buried in that table. For several rounds the comparison ran against "the Spark does 20 tok/s" — a figure sitting in my lab notes that my assistant kept quoting back at me, and that I kept accepting. It came from an engine that box stopped running in July. I have a dashboard that reports what every machine is actually serving, right now, and neither of us had opened it.

When we finally measured instead of remembering, the real figure was 14.75, and every conclusion built on 20 had been backwards. The junk rig hadn't been losing narrowly. It had been winning the whole time.

Takeaways

Where the time went. Not the build, and not the model itself. It went into two measurements that were confidently wrong: a benchmark configured to leave 85% of the VRAM empty, and a baseline quoted from memory instead of measured. Both produced clean, plausible, decision-shaped numbers. Both pointed the wrong way. The actual debugging — the MTP format archaeology, the tensor-split OOM — was fast by comparison, because those failures at least announced themselves.

Reusable diagnostics. Read the resource telemetry before you believe a benchmark's conclusion: 5 GiB used on a 22 GiB card is not a hardware ceiling. When something scales worse than expected, run the A/B that holds everything else fixed — the third card's 8.5% penalty is invisible in a sweep and obvious in a controlled pair. Read GGUF headers over HTTP range requests instead of downloading 86GB to check a tensor name. And when a feature "isn't supported," find out which layer says no: the architecture table, the loader, or the graph. Those are three different problems and only one of them is yours to fix.

The general principle. If a number drives a decision, re-measure it before you act on it. Both near-misses had the same root cause: I trusted a number that had been true once, in a configuration that no longer existed.

Conclusion

If you're running big MoE models on old hardware:

1. Never let -ncmoe sit at the block count while you interpret the result as a hardware limit. Check peak VRAM first.
2. Pass an explicit -sm layer -ts <split> for any multi-GPU offload. The default piles the back of the model onto the last card.
3. Make every capacity point generate real tokens. Loading is not running.
4. Test card count as its own variable, at a fixed CPU/GPU split, before buying anything.
5. For DeepSeek-V4-Flash specifically: MTP speculative decoding is blocked on llama.cpp implementing deepseek4 nextn support at all — not on your GPU, and not on finding the right GGUF.
6. Re-measure your baseline the day you use it to make a decision.

One card, 284B parameters, 262K of context, and two GPUs left over for something else.

---

Also in this series: Part 1 — Running a 119B MoE on three 2080 Tis · Part 2 — Laguna S 2.1, a 118B coding MoE on one card