[Benchmark] Qwen3.8-Flash-Next NVFP4 on a DGX Spark: 41.7 tok/s, 78% Over llama.cpp

A bolt-on modification is one you install in a driveway. Nine parts clip onto what is already there, and when you take them off the car is stock again: nothing cut, nothing welded, the factory never involved. The recipe in this article has that shape. It runs the stock vllm/vllm-openai:nightly image and bind-mounts nine Python files read-only over the container's own copies with -v host:container:ro. Unmount them, restart, and you are back on an unmodified nightly.

This is Part 45 of the DGX Spark series. Part 43 put the same model on this box under llama.cpp with the UD-Q4_K_XL quant: 20.51 tok/s on prose, and the full 262,144-token context inside the 121 GB pool. This part swaps the runtime underneath it. The recipe is tonyd2wild's, published at Qwen3.8-Flash-Next-NVFP4-DGX-Spark; I reproduced it on my own Spark at commit d83f10c (2026-09-05), matched the numbers, and wrote down what it costs to get there.

What you get: 41.7 tok/s median, 27.4 on prose, 47.2 aggregate at four streams

Look at the harness-median row first. That number decides whether 155 GB of disk and a 13-minute load are worth it. The rest of the table explains where it comes from.

measurement	my GX10	author's Spark	same-box llama.cpp seat
40-prompt harness median	41.7	43.9	23.38 (fleet probe)
prose	27.4	29.0	20.51 (Part 43)
coding	42.0–45.0	44.3	—
4 streams (per stream / aggregate)	26.1 / 47.2	26.4 / 47.1	—
32K prefill	1,666 tok/s, TTFT 17.3 s	—	—

MTP is multi-token prediction. A small draft head bolted onto the model guesses the next few tokens, and the main model verifies all of them in a single forward pass rather than one pass per token; when the guesses land, the box emits three tokens for roughly the memory traffic of one. That mechanism is why the same checkpoint moves from 17 tok/s to over 40 without a single weight changing. It is also why prose is the slowest row: a draft head has its lowest acceptance rate on free text, and Part 5 measured the same pattern on V4-Flash.

The 23.38 in the right-hand column deserves a caveat. It is a fleet probe: one generic prompt my monitoring sends to whatever answers on port 8000, not the 40-prompt harness, so treat it as an order-of-magnitude reference rather than a paired result. The paired number is prose, 27.4 here against 20.51 in Part 43, same box and same model. And llama.cpp still wins one workload outright: with n-gram speculation on a repetitive file edit it reached 76.65 tok/s, because a draft source that copies straight out of the prompt beats a learned draft head whenever the answer is mostly already on screen.

What you need: 155 GB of disk, 25 GiB of free memory, 13 minutes of patience

• One DGX Spark (GB10, 128 GB unified memory), tensor-parallel 1. The author's machine is the same class.
• About 155 GB of free disk: 124 GiB (133 GB) for the checkpoint plus the 22.2 GB container image.
• Docker with the NVIDIA container toolkit. No CUDA toolkit on the host, no compiler, no vLLM build.
• At least 25 GiB of MemAvailable before launch, and a drop_caches immediately before it. The load bottoms out at 25.6 GiB.
• 12 minutes 54 seconds of cold start: 585.81 s for the weights, another 69.16 s for the MTP draft head.

One property of the model explains those memory numbers. Of its roughly 180B parameters (125B main model, 51B PLE, 4B MTP head), 51B are a PLE, a per-layer embedding table that each token reads a few rows out of, closer to a dictionary lookup than to a matrix multiply. The launcher's default staged mode leaves that table on NVMe and gathers only the rows each forward pass needs, so the resident weights come to 76.48 GiB instead of the full 124 GiB on disk.

Do it in this order: clone, download, pull, copy, launch

git clone https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark ~/src/tonyd2wild-nvfp4
cd ~/src/tonyd2wild-nvfp4 && git log -1 --format='%h %ad' --date=short   # d83f10c 2026-09-05
hf download nvidia/Qwen3.8-Flash-Next-NVFP4 --local-dir /home/coolthor/models/qwen38-fn-nvfp4-nvidia
docker pull vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c
mkdir -p ~/patches && cp -r single-spark-vllm-tp1/patch ~/patches/qwen4exp-ple-mmap   # the launcher's default PATCH_DIR
cp single-spark-vllm-tp1/launch/qwen38fn-nvidia-tp1.sh ~/patches/
PORT=8000 NAME=qwen38fn bash ~/patches/qwen38fn-nvidia-tp1.sh
docker logs -f qwen38fn 2>&1 | grep -m1 'Application startup complete'   # ~13 min

The launcher needs no environment variables for the fast path, because its defaults are the author's 2026-09-05 single-node best. PLE_MODE=staged keeps the 51B table on NVMe. GRAPHS=nocompile captures CUDA graphs for decode at batch sizes 4 through 24 and skips torch.compile entirely. MTP=3 drafts three speculative tokens per step, and in practice around three or more get accepted. DRAFT_VOCAB=65536 restricts the draft head to the lowest 65,536 token ids, using BPE merge order as a stand-in for frequency. It is an FR-Spec idea, and the author credits MiaAI-Lab with applying it to this model first. The remainder are ordinary serving knobs: --max-num-seqs 6, --max-num-batched-tokens 4096, --kv-cache-dtype fp8_e4m3, --gpu-memory-utilization 0.80, --max-model-len 262144. Prefix caching is off and stays off. The author cites vLLM issue #54173, a GDN-layer crash with prefix caching enabled, marks it as unverified, and I did not verify it either. Every number in this article was taken without prefix cache.

The nine overlays split three ways. Four are the author's own PLE and MTP implementation: ple_layer.py, ple_mmap.py, model_state.py, and mtp_draft_vocab.py, which lands on the container's mtp.py. Four are unmodified upstream PR code — ops_ple.py from PR #55375 by peakcrosser7, merged 2026-09-05, which fixes a fused-PLE conv state-index stride for MTP and for concurrent prefills; plus ops_qsa.py, qsa.py, and platforms_interface.py from PR #54846 by andreasgru, still open, which puts fp8_e4m3 and NVFP4 KV cache on the QSA path. The ninth is modelopt.py, an upstream file carrying two fixes of the author's own: draft-local MTP index candidates, so that vLLM's mtp.layers.48 finds ModelOpt's mtp.layers.0 quant entry, and a route from FP8_BLOCK_SCALES routed experts to Fp8MoEMethod with its 128×128 block scales. The author notes that sfxnz (MIT) and MiaAI-Lab (AGPL) closed the same two gaps independently on the same day, with no shared code.

Once Application startup complete appears, the smoke test is one request. Send stream_options even if you do not think you need it; the first pitfall below is what happens when you leave it out.

curl -s http://<your-gx10-ip>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen38-flash-next",
       "messages":[{"role":"user","content":"Say hello in one sentence."}],
       "temperature":0, "max_tokens":200,
       "stream":true, "stream_options":{"include_usage":true}}'

Four pitfalls, one of which had been hiding for two months

vLLM streams without usage counts unless you ask for them. A streaming response carries no token accounting unless the request includes stream_options: {"include_usage": true}. llama-server attaches usage on its own, so any tok/s script ported over from llama-server divides by nothing on its first run and reports nothing at all.

vLLM validates the model field; llama-server answers to any name. The night I switched, the container log filled with 404s from two scripts on my laptop that were still sending an old model id. They had been broken for two months and nobody noticed, because llama-server replies to whatever string you hand it. Audit every client's model name before you switch runtimes, not after.

--network host takes port 8000 outright. The container binds the port directly, so the existing seat has to be stopped before the new one starts. The production seat was down for 33 minutes, and that figure includes the 13-minute load plus every acceptance test.

Prefix caching is off by design. The launcher disables it because of the crash described above, so a repeated system prompt is re-evaluated every request. If your workload leans on long shared prefixes, budget prefill accordingly.

Deep dive: the morning I calculated "even fixed, it's only 23"

Skip this section if you only want the recipe — everything needed to run it is above. What follows is the debugging record, including the decision that told me to stop, a day before the numbers said otherwise.

Episode 1 — 2026-09-05: without MTP, the floor is 17 tok/s

Problem. I wanted a number for how fast NVIDIA's NVFP4 checkpoint runs on a GB10. One community benchmark (jschmied's GB10 ladder) had this checkpoint at 36.5 tok/s, but on a different runtime combination, so it was context rather than a target I could aim at.

What I expected. NVFP4 weights are smaller than UD-Q4_K_XL on the same memory bandwidth, so the floor should land at or above the llama.cpp seat's 23.38.

What I did. Built my own vLLM image with PLE mmap and measured it: batch size 1, temperature 0, thinking off, a fresh nonce per run to defeat caching, max_tokens 800, three runs of prose and three of coding.

Result. 16.62 to 17.01 tok/s, and prose ran at the same speed as coding. Effective weight traffic came to 9.27 GiB per token, which works out to 154–158 GiB/s (165–169 GB/s), about 61% of the 273 GB/s nominal. That is 28.6% slower than the llama.cpp seat, not faster.

What the expectation missed. Single-stream decode is bounded by memory passes per token, not by how small the weights are; a smaller format that still walks the same layers once per token buys nothing. Part 32 had already measured NVFP4 as compression rather than compute on this box, and Part 42 put the practical bandwidth ceiling around 85% of nominal; a 61% floor with no MTP fits both. The flat prose-versus-coding result was the second tell and I underweighted it. On the 27B model on another box, prose and coding differ by roughly 2×, and that spread comes entirely from MTP acceptance rate. No spread means no MTP was running.

Episode 2 — 2026-09-06, early morning: MTP k=2 dies during weight load

Problem. Turning MTP on with --speculative-config '{"method":"mtp","num_speculative_tokens":2}' killed the server before it finished loading weights.

AttributeError: Layer mtp.layers.48.mlp.experts has no parameter
'w2_weight_scale_inv' for checkpoint weight
'mtp.layers.48.mlp.experts.0.down_proj.weight_scale_inv'

What I expected. A memory ceiling or a wrong flag. Adjust one thing, restart, move on.

What I did. Read the checkpoint's hf_quant_config.json alongside vLLM's qwen4_exp/nvidia/mtp.py and modelopt.py, following the key that appears in the error message back to where it should have been registered.

Result. Two independent gaps, not one. First, the checkpoint quantizes the MTP experts as FP8_BLOCK_SCALES with group size 128, under the key mtp.layers.0.mlp.experts, while vLLM numbers that layer mtp.layers.48; the draft config remaps ignored_layers and exclude_modules but never quantized_layers, so the quant entry is never found. Second, even with the prefix corrected, the RoutedExperts branch in modelopt.py handles FP8, NVFP4, W4A16_NVFP4 and MXFP8, and FP8_BLOCK_SCALES appears nowhere in that tree. I estimated the fix at three files and 80 to 155 lines.

What the expectation missed. Nothing, as it turned out. The diagnosis held up: the author's modelopt.py overlay patches exactly those two spots. The bad call came an hour later.

Episode 3 — 2026-09-06, morning: the estimate said "about 23 even if fixed", so I stopped

Problem. Whether to spend the day patching vLLM's ModelOpt parser to get MTP running at all.

What I expected. The community numbers put MTP at roughly +35%, so I applied it to my measured floor: 17.0 × 1.35 ≈ 23, which is parity with the llama.cpp seat's 23.38. An 80-line patch to arrive exactly where I already was does not clear the bar. I made the call to pivot and moved the MTP effort to the llama.cpp side.

What I did. Wrote the decision down. No ticket, no download, no seat change, and I went to work on something else.

Result. That same afternoon I read tonyd2wild's repo: same checkpoint, same class of hardware, eager mode with MTP3 at 35.2 tok/s and the full recipe at 43.9. That evening my own box produced 41.7.

What the expectation missed. The formula had exactly one multiplier in it, "MTP ≈ +35%", a figure someone else measured on a 26.1 baseline in a different configuration, which I treated as a property of MTP itself. What it left out: MTP3 accepting around three or more tokens per step, a draft head restricted to 65,536 token ids, the PLE table moved to disk to free memory, and decode CUDA graphs captured without torch.compile. The only instrument that can veto an experiment is another experiment.

Episode 4 — 2026-09-06, evening: a 2-12% reproduction gap I have not explained

Problem. Every single-stream category came in under the author's: prose 27.4 against 29.0 (−5.5%), JSON 44.3 against 49.3 (−10%), summary 31.0 against 35.4 (−12%). Concurrency, meanwhile, matched almost exactly: 26.1 / 47.2 against his 26.4 / 47.1.

What I expected. Same image, same overlays, same harness, same checkpoint, therefore within ±3%.

What I did. Compared the two result sets once. No reruns.

Result. The gap is real and its cause is unverified. Three explanations are still open. His environment may differ from mine; his log mentions three reruns after power loss. My box runs earlyoom plus a set of resident services. And five prompts per category is a small sample. The one constraint I have is that the concurrent numbers agree, which argues against a GPU-level or memory-bandwidth difference, since that would show up in the aggregate too.

What the expectation missed. I treated single-stream and concurrent numbers as equally reliable, when single-stream decode is far more sensitive to scheduling jitter from anything else on the machine. Explaining this needs at least three runs per category, and that is a follow-up, not a claim this article makes.

Full parameters: the exact command behind every number

The launcher wraps this, but here is what it expands to, including the cache flush the author's README asks for on every start.

# Before every start (author's README):
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

docker run --gpus all -d --restart no --network host --ipc host \
  --shm-size 32g --ulimit memlock=-1:-1 --name qwen38fn \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e FLASHINFER_CUDA_ARCH_LIST=12.1a -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e VLLM_USE_DEEP_GEMM=0 -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e QWEN4EXP_PLE_MMAP=1 -e QWEN4EXP_PLE_STAGED=1 \
  -e QWEN4EXP_PLE_MMAP_THREADS=64 -e QWEN4EXP_DRAFT_VOCAB=65536 \
  -v /home/coolthor/models/qwen38-fn-nvfp4-nvidia:/models/qwen38fn:ro \
  -v /home/coolthor/qwen38fn-vllm-cache:/root/.cache \
  -v $P/ple_layer.py:$VP/models/qwen4_exp/nvidia/ple_layer.py:ro \
  -v $P/ple_mmap.py:$VP/models/qwen4_exp/nvidia/ops/ple_mmap.py:ro \
  -v $P/model_state.py:$VP/models/qwen4_exp/nvidia/model_state.py:ro \
  -v $P/mtp_draft_vocab.py:$VP/models/qwen4_exp/nvidia/mtp.py:ro \
  -v $P/upstream-overlays/ops_ple.py:$VP/models/qwen4_exp/nvidia/ops/ple.py:ro \
  -v $P/upstream-overlays/ops_qsa.py:$VP/models/qwen4_exp/nvidia/ops/qsa.py:ro \
  -v $P/upstream-overlays/qsa.py:$VP/models/qwen4_exp/nvidia/qsa.py:ro \
  -v $P/upstream-overlays/platforms_interface.py:$VP/platforms/interface.py:ro \
  -v $P/upstream-overlays/modelopt.py:$VP/model_executor/layers/quantization/modelopt.py:ro \
  vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c \
  --model /models/qwen38fn \
  --served-model-name qwen38-flash-next qwen3.8-flash-next \
  --quantization modelopt --trust-remote-code \
  --max-model-len 262144 --max-num-seqs 6 --max-num-batched-tokens 4096 \
  --kv-cache-dtype fp8_e4m3 --gpu-memory-utilization 0.80 \
  --no-enable-prefix-caching --no-enable-flashinfer-autotune \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[4,8,12,16,20,24]}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

# $P  = ~/patches/qwen4exp-ple-mmap                   (host: the nine overlay files)
# $VP = /usr/local/lib/python3.12/dist-packages/vllm  (container: the vLLM package root)

Cold start ran 12 minutes 54 seconds end to end: 585.81 s for the weights and 69.16 s for the MTP draft head. The vLLM log reported Model loading took 76.48 GiB, and the KV cache allocated alongside it holds 1,130,661 tokens, which is 4.31× the 262,144-token context. MemAvailable bottomed at 25.6 GiB during the load, settled at 14.3-14.9 GB in steady state, and dipped to 12.9 GiB during the 32K prefill. About 4 GB of swap was in use throughout and earlyoom, configured at -m 6, never fired. Short-prompt TTFT sat between 0.21 and 0.28 s. The 32K prefill took 28,903 input tokens in 17.3 s, about 1,670 tok/s, and the needle I planted mid-prompt, PELICAN-7, came back. Vision and tool calling both work off this same server: a 64×64 red PNG returned "Red", and asked about Taipei weather with a get_weather tool schema, the model returned {"city":"Taipei"} and finish_reason: tool_calls.

Across all eight harness categories my box lands below the author's on every single-stream measurement and level with him on concurrency, which is the pattern Episode 4 covers.

category	my GX10	author's Spark
prose	27.4	29.0
coding	42.0–45.0	44.3
reasoning	44.6	45.6
JSON	44.3	49.3
HTML	46.8	47.6
narrative	27.7	28.5
summary	31.0	35.4
formatting	30.3	32.4
harness median	41.7	43.9
4 streams (per stream / aggregate)	26.1 / 47.2	26.4 / 47.1
6 streams (aggregate)	not run	68.8
eager mode + MTP3, no CUDA graphs	not run	35.2

There is a faster path, and I have not taken it yet. MiaAI-Lab re-quantized the same model into their own 99 GiB (106 GB) checkpoint and report 46.3 tok/s single-stream on prose, 108.1 aggregate across four streams, and 2,265 tok/s on a 32K prefill. It does not rebuild vLLM either; its start script extracts the pristine sources from the image and patches them at launch. What it costs is another 106 GB download, and the recipe repo is AGPL-3.0, while the checkpoint itself is Apache-2.0. My order of operations was to get the NVIDIA path stable on the production seat first, so that one is parked.

Takeaways

Where the time actually went

Not the run. Installing the recipe is bounded by how fast you can pull 133 GB, and the box was back in service 33 minutes after I stopped the old seat, load and acceptance tests included. Half a day went into the estimate in Episode 3: one multiplier, one decision, no ticket. Overturning it took reading one README and waiting 13 minutes for a load.

The 41.7 is the smaller of the two findings here. The larger one is the shape of the estimate that nearly buried it: a formula assembled only from mechanisms I already knew about will keep answering "no headroom", and it will be most confident precisely when the headroom is somewhere I have not looked.