[Local LLM] Depth-1 MTP on V4-Flash: +9% on agent turns, −4% on prose — route speculative decode by workload

The short version

Speculative decoding is a helper that tries to finish your sentence. If it guesses right, you got two words out for the price of one; if it guesses wrong, you wasted the guess and have to say the word yourself anyway. Whether that's a net win depends entirely on how guessable your sentence is.

That's the whole story of this post. On my GB10, checking a guess (the "verify" step) costs ~108ms; making the guess (the "draft") costs ~4ms. So the guess is almost free — the expensive part is checking it. When the model is writing code, JSON, or tool calls — output where the next token is highly predictable — the guess is usually right, and you bank a nearly-free extra token per expensive check. That's the +6 to +9%. When it's writing free prose or chatting in Chinese — output where the next token is genuinely open — the guess is more often wrong, so you paid the small draft cost for a token you throw away. That's the −4%.

So depth-1 MTP is not a "make it faster" toggle. It's a trade whose sign flips with the workload. Enable it for the structured stuff, and for everyday chat either accept the small tax or send that traffic down a different path.

---

The setup

Depth-1 MTP (ds4 PR #371, "depth-1 continuous") is the modest version of speculative decode: one drafted token, one verify. It's the always-on default a lot of people leave enabled without checking. I checked. The answer is that it wins — but only on some workloads, and the ones it loses on are exactly the ones you'd reach for it on if you were guessing blind.

All numbers below are my own measurements on a single 128GB GB10 running the ds4 CUDA build, and the output is lossless — byte-identical with and without the draft, verified. The draft changes how fast the tokens come out, never which tokens they are.

The one-line result: the sign follows the workload

Workload	MTP-1 vs no draft
Agent / tool-use	+9.4%
Code	+6.5%
Technical text	+4.4%
Free prose	−3.6%
Chinese chat	−3.7%

Same model, same box, same flag — the only thing that changed between rows is what I asked it to generate. Structured, predictable output gets faster; open-ended output gets slower. The spread from best to worst is 13 points, and it straddles zero. That last part is what makes this interesting: there is no single true "MTP-1 speedup" number for this model. The honest answer to "how much faster is speculative decode?" is "on what?"

Why the sign flips: decode here is verify-bound

Depth-1 spec decode does two things per step: draft one token ahead (~4ms), then run one verify pass (~108ms) that either accepts or rejects it. The verify dominates by ~27×. That asymmetry is the entire mechanism.

• Draft accepted: the verify pass commits two tokens (the base token plus the drafted one) instead of one. You got a second token for essentially the same 108ms. Net win.
• Draft rejected: the verify pass commits one token, and the 4ms you spent drafting is gone. Net loss — small, because 4ms is small, but a loss.

So the sign of the speedup is just the sign of (free tokens from accepts) minus (wasted drafts from rejects). And whether a draft gets accepted is a property of the workload: when the next token is highly predictable — the closing brace after a block, the argument name in a known tool schema, the next keyword in boilerplate — the draft nails it and you bank the free token. When the next token is genuinely open — the next word in a sentence, the next turn in a chat — the draft is a coin flip, rejects pile up, and the wasted 4ms drafts outweigh the wins.

That's why the ranking is exactly agent > code > technical text > prose > chat: it's a predictability ranking. The speedup didn't rank the workloads; the acceptance rate did, and the acceptance rate is just how guessable the text is.

The limiter isn't draft quality — it's a margin gate

Here's the part that surprised me. Across 231 speculative cycles I logged, the average committed τ (tokens per cycle) was ≈1.54 — so on average each cycle banks about half an extra token. You might read a soft number like 1.54 as "the drafter is only so-so." It isn't. When a cycle actually runs its verify, ~97% of drafts are accepted. The draft quality is excellent.

What holds τ down is that a margin gate skips the verify entirely on ~44% of cycles — when the model's own confidence margin is too thin to bother speculating, it declines to draft-and-verify and just decodes normally. So the ceiling on MTP-1 here isn't "the drafter guesses wrong" (it almost never does, once it commits); it's "the drafter only plays the hand when it's confident, and it folds 44% of the time." That's a conservative design — it's why the losses on prose are only −3.6% and not −15%: the gate refuses most of the bad bets before they cost a verify.

It also means there isn't much headroom to tune on depth-1. You can't make a 97%-accept drafter much better. The lever that would move τ is drafting more tokens per verify — a bigger, separate bet, which I get into below.

Why not draft more than one token?

The natural next thought is "if one nearly-free drafted token helps, draft five and amortize the 108ms verify across all five." It's a real option, and a bigger bet. The catch is that a multi-token block only pays if the whole block survives verification: one rejected token collapses the amortization, and the probability that all N survive falls off fast as the text gets less predictable. A block draft is really a bet that the model's next several tokens are all guessable — which is a much stronger claim than "the next one is," and it gets shakier the longer the block. That's its own investigation with its own tradeoffs (and its own post).

Depth-1 is the floor: the smallest possible bet, so it's the one most likely to clear the acceptance bar — and on predictable workloads it clears it 97% of the time. This post is about making that floor pay, which turns out to be entirely a routing question.

How to route it: on for code/agent, route chat elsewhere

The practical takeaway is that "speculative decode: on/off" is the wrong granularity. Route it:

• Code, agent, tool-use → MTP-1 on. These are your +4 to +9% workloads, and they're also the ones where latency matters most — an agent turn is a chain of tool calls, each one a round trip you're waiting on. This is the free money.
• Everyday chat / creative → don't pay the −4%. Two options. Accept it — 3.6% is small and you may not want two serving paths. Or route that traffic to the non-thinking deepseek-chat path from Part 2, which is a better fit for casual turns anyway and sidesteps the wasted-draft cost entirely.

The meta-point: a speculative decoder is a bet on predictability, and you already know which of your workloads are predictable. You don't have to measure per request — code and tool calls are structured, chat and prose aren't. Set the flag by workload class, not globally.

Takeaways

Where the time went

Not in the tuning — depth-1 MTP is one flag. It went into not stopping at the average. A single blended "MTP-1 is +2%" number would have been true and useless; it hides a +9.4% and a −3.7% that cancel. The time went into slicing the benchmark by workload until the sign flip showed up, because the sign flip is the whole finding. An average speedup across mixed workloads is a number that describes no real workload you actually run.

Reusable diagnostics

When a speculative-decode result is small and positive, don't trust the average — split it by how predictable the output is and look for a sign flip. The tell for "this is verify-bound and acceptance-gated" is the ratio of verify cost to draft cost (here ~108ms vs ~4ms = 27×) together with the accept rate on committed cycles (~97%). If verify dwarfs draft and committed drafts almost always accept, your limiter is the gate that decides whether to speculate, not the drafter's quality — and your speedup will track workload predictability, not any flag you can tune.

The general principle

Speculative decoding is not a global "faster" switch — it's a workload-dependent trade whose sign follows the acceptance rate. Depth-1 wins where acceptance is high and loses where it's low, by roughly the same small margin. Enable it where your output is predictable, and know it's costing you a little where it isn't.

Conclusion: the checklist

1. Turn MTP-1 on for code, agent, and tool-use. +4 to +9% on exactly the latency-sensitive workloads, lossless.
2. Don't turn it on globally. Free prose and chat pay −4%; the speedup's sign follows how predictable the output is.
3. Route chat elsewhere. Either eat the small cost or send casual turns to the non-thinking deepseek-chat path (Part 2).
4. Don't try to tune depth-1. Committed drafts already accept ~97%; the limiter is a margin gate that folds 44% of cycles, not draft quality.
5. Don't reach straight for a bigger block to "do better." Drafting more tokens per verify is a much stronger bet on predictability, and the compounding rejection risk is real — measure it before assuming it wins.

---

Also in this series: Part 2 — Running a 15 tok/s 284B as your daily agent brain: the settings that make it bearable · Part 4 — My 284B agent quietly stopped reusing its KV cache: the ds4 evict storm · Upstream: ds4 PR #371