[DeepSeek-V4-Flash] Swapping the ds4 Engine for a Free Half-Generation Speedup on One DGX Spark

The short version: a faster engine is not a different model

Here's the idea for a friend who doesn't do this for a living. The thing sitting on my desk runs a large AI model — DeepSeek-V4-Flash, which I use as the brain for my whole home agent setup. There are two separate pieces at play: the model (the weights — the actual learned "knowledge") and the engine (the program that loads those weights and runs the math to produce words). People obsess over the model. But the engine is just software, and someone can rewrite it to run the same model faster on the same chip. That's what happened here. I didn't get a smarter model. I got the same one, running about a third faster, by swapping the program underneath it and touching nothing else.

The one-question detour that found a faster engine

I wasn't looking for a speedup. I run ds4 (DeepSeek-V4-Flash, a Mixture-of-Experts model where only a few of its many sub-networks fire per token) on a single DGX Spark, and it decodes at 14-16 tok/s. I'd quietly filed that under "the ceiling" and moved on.

Then, out of pure curiosity, I asked Codex one question: does this ds4 repo have any optimization specific to a single GX10? It came back with Entrpi — a Blackwell CUDA performance fork of antirez's ds4, tuned specifically for this box. This is a short entry in my ongoing "DeepSeek-V4-Flash on DGX Spark" series; earlier parts squeezed the model, and this one is about squeezing the engine instead.

Goal: the same abliterated model, just faster

I wanted to be explicit about the constraint, because it's the whole reason this wasn't a simple "download the new thing" story. I don't run stock ds4. I run an abliterated huihui variant — a version with the model's built-in refusals surgically removed — because it's the brain of my agent stack and I don't want it declining tool calls or moralizing at me mid-task.

So the goal was never "get a better model." It was "get the exact same abliterated model to run faster." Any speedup that required switching back to the refusing standard version was, for me, not a speedup at all.

The payoff: same abliterated model, decode 14-16→20, prefill 2×

Entrpi's speedups are engine-level, not tied to any particular model: rewritten tensor-core prefill kernels (the fast matrix-multiply units on the GPU), decode wrapped in CUDA graphs (pre-recorded GPU command sequences that cut per-step launch overhead), and a compressed KV cache. None of that cares whether you load the standard or the abliterated V4-Flash weights. So I kept the abliterated huihui completely unchanged and just swapped the binary underneath it.

The numbers, measured on my one DGX Spark:

• Decode: 14-16 → about 20 tok/s (two runs, 20.0 and 20.8).
• Prefill: roughly 2× — 722 tok/s on a 5511-token prompt. That one matters more than it looks, because my system prompts are long, and prefill is the model reading that whole prompt before it emits a single word. Halving that wait is felt on every single turn.

Same brain, faster body, and not one bit of the model changed. A free speedup like that is not one you leave on the table.

How: the swap took 10 minutes, the safety net took the evening

The swap itself is boring. Point the install at Entrpi, load the same GGUF, restart. Ten minutes. The interesting part was everything around it.

Read the file that runs, not the file that reads. Codex told me to install v0.3.0. The README said v0.2.2. For about a minute I assumed Codex had hallucinated a version. Then I opened the thing that actually executes — install.sh — and it pins v0.3.0. The README had simply lagged behind the code. Lesson, filed permanently: verify the version from the file that gets executed (install.sh), not the one that gets read (README).

The production safety net, because I couldn't A/B. 128GB of unified memory (one pool the CPU and GPU share) can't hold two models of about 80GB each, so there was no running the old and new engines side by side. To load Entrpi I had to stop the live service on :8000 — the one my agents actually talk to. And before I even got there, two dumb bugs kept the new engine from starting:

# bug (a): grab the main model — except this grabbed the MTP draft file
MODEL=$(ls *.gguf | head -1)   # -> ds4-MTP-draft.gguf. wrong file.

# bug (b): $HOME expands on MY laptop, not on the box
ssh spark "cd $HOME/ds4 && ./serve.sh"   # -> /Users/coolthor, not /home/coolthor

Both failed identically from the outside: "won't start." What saved me was the order of operations in my deploy script. It saved the live startup command first, then killed the service, with a trap that restores the old one on any error:

OLD_CMD=$(cat /home/coolthor/ds4/serve.cmd)   # capture BEFORE killing
trap 'echo "startup failed — restoring :8000"; eval "$OLD_CMD"' ERR

So both times the new engine face-planted, :8000 came back on its own in about 5 minutes — no manual rescue, no scramble. In production — someone else's or your own — a safety net isn't a bonus feature. It's a prerequisite for being allowed near the thing at all.

Debug 1: speculative decoding gets slower on an abliterated model

Once I had 20 tok/s I got greedy and reached for speculative decoding — a trick where a small, cheap "drafter" model guesses the next few tokens, and the big model verifies them all in parallel, keeping the ones it agrees with. When the guesses are good, you get several tokens for the price of one verification step. Entrpi ships two drafters: MTP (multi-token prediction) and DSpark. I tried all of them. They all lost.

The last column is the tell — it's how many tokens the big model actually accepted per verification step. 1.0 means the drafter bought you nothing.

Config	decode	tokens/step
plain (no speculation)	about 20 tok/s	1.00
MTP draft-1	20.1 (tie)	1.62
MTP draft-2	15.8 (slower)	1.57
MTP draft-3	17.1 (slower)	1.61
DSpark	16.5 (slower)	1.11

Look at DSpark: 1.11 accepted tokens per step means the big model rejected nearly every guess. And here's why, and it's the whole point of this section: every one of these drafters was trained for the standard DeepSeek-V4-Flash. Abliteration shifted my model's behavior distribution, so the drafter is confidently predicting tokens my model no longer produces. Every rejected guess is wasted work, and that overhead drags a clean 20 tok/s down to 16.5. MTP tells the same story from the other side — a deeper draft is more speculative work per step, but acceptance doesn't climb to pay for it, so both deeper settings land below plain decode.

The only real path to a speculation win on an abliterated model is to train your own drafter matched to that abliterated model. I priced it out: tens of terabytes of teacher-model cache, a multi-GPU node to train on, and an undocumented PyTorch-to-GGUF conversion at the end — all for roughly a 30% gain I'm not even sure survives. Not worth it. So on the abliterated model, the verdict is blunt: turn speculation off. The plain engine is the fastest thing I've got.

Debug 2: disk-KV doesn't save memory, and managed KV isn't swap

Separately, I wanted to push context from 256K to 512K tokens — and ran straight into two things I thought were true and had to disprove by measuring.

"disk-KV saves memory" — actually it's checkpoint restore. I'd dropped the old --kv-disk-dir flag somewhere in the shuffle. I re-added it, opened 512K, and the boot log still said it was using managed KV. That's when it clicked: ds4's disk-KV is not live-KV offload. It saves a long prompt's stable prefix (its KV cache — the model's cached read of the tokens so far) to disk, so the next time the identical prompt is resent it restores in 87ms instead of recomputing — fantastic for an agent that reuses a fixed system prompt every turn. But it does nothing to relieve memory pressure: the live KV for a 512K context still has to fit in the 128GB.

"managed KV means it's swapping" — actually it wasn't swapping. At 512K the KV cache is about 11.3GB (256K is 5.9GB). On first start the native path OOM'd, and ds4 honestly fell back to managed KV cache, with a boot-log warning that it "may degrade performance." I read "degrade" as "now swapping to disk, this is going to be miserable." Then I actually looked:

system swap used:        464Mi
ds4 process swapped-out:  75MiB

That is not a process that's swapping. Managed KV is CUDA unified managed memory — memory the driver auto-pages between GPU and system RAM and lets you oversubscribe. The "degrade" warning is the cost of that paging management, not disk swap. Real swap only kicks in if you genuinely fill 512K past physical memory. So I kept 512K: normal use still decodes the same 20 tok/s with no swap, and only a genuinely enormous context would ever touch the managed edge. Context doubled, at zero cost.

Takeaways

Where the time went. Not the engine swap — that was 10 minutes. It went into the gap between "I thought" and "actually." Three assumptions — speculation will help, disk-KV frees memory, managed KV means swapping — each felt obviously true and each was wrong, and the only way through was to measure them into the ground. The measuring was the work.

Reusable diagnostics. Separate a theoretical risk from the actual state, and let the numbers arbitrate. "Managed KV might swap" is a theory; 464Mi of measured swap is the actual state, and it said fine. The log's wording ("may degrade performance") was the theory dressed up as a fact — I almost trusted the sentence instead of reading the counter next to it. When a tool warns you about a failure mode, go measure whether that failure mode is actually happening before you act on the warning.

The general principle. Everyone chases a bigger model. But swapping the engine is often a free half-generation upgrade hiding in plain sight: same weights, same chip, a faster program — and here that bought +30% decode, 2× prefill, and zero change to the model I actually wanted to keep. Before you go shopping for more parameters, check whether someone already rewrote the thing running the ones you have.

The checklist: swapping an inference engine without a live A/B

1. Verify the version from the file that executes. install.sh pins the truth; the README lags. When a summary and a doc disagree, open the script.
2. Save the live command, then kill — with a restore trap. With one memory pool you can't run old and new side by side, so make failure auto-recover. Both of my deploy bugs cost me zero manual recovery because :8000 restored itself.
3. On an abliterated model, leave speculation off. Stock drafters are trained for the standard model and mispredict the shifted distribution; measured acceptance was as low as 1.11 tokens/step, turning a 20 tok/s engine into 16.5. A matched custom drafter is the only win, and it wasn't worth the cost here.
4. Measure swap before believing the "degrade" warning. disk-KV is checkpoint restore, not offload; managed KV is unified managed memory, not disk swap. At 512K I measured 464Mi system swap — doubling the context cost me nothing.

That wraps up this part of the "DeepSeek-V4-Flash on DGX Spark" series — a free half-generation upgrade for the price of one idle question to Codex. The engine is Entrpi/ds4-on-spark, a Blackwell CUDA fork of antirez/ds4; both are doing the heavy lifting here.