[Benchmark] Two Modded 2080 Tis Reach 59.6 tok/s on Qwen3.8-27B With llama.cpp Tensor Parallel

Two people carrying a long table are slower than one person dragging it, unless somebody calls the cadence. The extra pair of hands was never the hard part; the calling was. Two GPUs running one model work the same way: bolting in the second card takes ten minutes, and getting the two of them to agree with each other on every layer is the rest of the story.

This is Part 14 of the modded-2080-Ti series. Part 13 got one card from 41.6 to 47.6 tok/s by fixing a broken chat template and retuning the draft depth on Qwen3.8's built-in MTP head. This part adds a second card and asks a different question: does splitting one model across two GPUs actually help, or does it just move the bottleneck somewhere else?

Short answer: it helps, but only on the second attempt, using a split mode most older guides don't mention, with an environment variable that isn't in anyone's example command.

What you need

• Two GPUs with enough combined VRAM, ideally matched. Mine are both modded RTX 2080 Ti 22GB cards, Turing, compute capability sm_75, in one EPYC box.
• A recent CUDA build of llama.cpp with NCCL support. I used b10064.
• Qwen3.8-27B as a Q4_K GGUF, with its built-in MTP (multi-token prediction) draft head: a small extra head baked into the model that guesses several tokens ahead so the big model can verify them in a single forward pass instead of generating them one at a time.
• Patience for one environment variable nobody puts in their example command. Covered below.

The production config

CUDA_VISIBLE_DEVICES=0,1 \
GGML_CUDA_ALLREDUCE=internal \
llama.cpp-b10064/build-cuda75-nccl/bin/llama-server \
  -m /path/to/Qwen3.8-27B-Q4_K.gguf \
  -sm tensor -fa on -c 262144 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --mmproj /path/to/mmproj.gguf

Four things in that command are doing more work than their length suggests:

flag / var	what it does
GGML_CUDA_ALLREDUCE=internal	picks the PCIe allreduce path over NCCL. An allreduce is the step that combines partial results computed on separate GPUs back into one answer, needed constantly once a model's math, not just its layers, is split across cards. Leave this unset and NCCL is the default on Linux, and NCCL is the one that didn't work for me (deep dive below)
-sm tensor	split mode: the flag that decides how a model's weights get divided across GPUs. tensor slices individual weight matrices so each card computes a piece of every layer's math, instead of giving each card whole layers to itself
--spec-draft-n-max 3	draft depth for MTP. 3 beat 2, 4, 5 and 7 (n=6 untested); not the depth that won on one card
--mmproj	the vision projector GGUF. I didn't need vision for this workload, but I loaded it anyway and ran a smoke test: asked the model the colors in a three-color-block PNG, and got red, green, and blue back correctly

I did not pass -ctk or -ctv. The main KV cache stays at f16 by default, and that default turns out to matter quite a bit (also in the deep dive).

37 to 59.6 tok/s: what two cards plus MTP actually buys you

Decode speed on the same coding prompt, four configurations, each one changing exactly one thing from the row above it:

Setup	decode tok/s (code)	vs. one-card baseline
1 card, Q4_K + MTP n=4	37.1016	baseline
2 cards, -sm row (build b9870)	38.4308	+3.6% (noise)
2 cards, -sm tensor, no MTP	37.5449	+1.2% (nothing)
2 cards, -sm tensor + MTP n=3, f16 KV	59.632	+60.7%

The takeaway from that table is simple: splitting the model alone gets you nothing. Both middle rows sit within measurement noise of the single-card number. Whatever GPU you add, if the draft head isn't also running across it, you paid for a second card to sit there.

The 59.632 tok/s figure is a mean of two short-prompt runs: 61.339 tok/s with 281 of 352 drafted tokens accepted, and 57.925 tok/s with 274 of 373 accepted. On the same config with a real 21,201-token prompt filling the context first, prefill ran at 776.805 tok/s and decode settled to 48.989 tok/s with 258 of 420 tokens accepted, slower than the short-prompt number as expected with more context to attend over, but nowhere near the single-card baseline. Traditional-Chinese prose decode came in at 34.4 tok/s on the same config, against 18.6074 on one card.

Final VRAM per card: 18,888 MiB and 17,738 MiB. For reference, the one-card baseline's own acceptance rate already sat at 68.22% and 70.17% on code, 21.92% and 22.86% on prose. MTP was doing real work before the second card entered the picture at all. Tensor parallel didn't invent that mechanism. It just gave it more headroom to run in.

Within 10-22% of a current mid-range pair, for a fraction of the price

As of 2026-08-22, I couldn't find a published paired benchmark of Qwen3.8-27B on dual 2080 Tis using llama.cpp tensor parallel plus MTP. That's a narrow claim on purpose: it means I didn't find one, not that one doesn't exist.

What is published is a fair comparison target. A community table for this same model reports 65.9 tok/s at draft depth 2 and 76.0 tok/s at draft depth 3 on two RTX 5060 Tis:

Setup	decode tok/s
2× RTX 5060 Ti, n=2 (community)	65.9
2× RTX 5060 Ti, n=3 (community)	76.0
2× modded 2080 Ti, n=3 (this run)	59.632

That card is current-generation. Mine is a 2018 flagship with a factory VRAM mod. It puts a pair of five-year-old secondhand cards within shouting distance of a current mid-range pair, about 22% behind their n=3 figure and 10% behind their n=2 one, for a fraction of the price: each of mine cost ¥2079 on Taobao and landed around NT$11,000 with shipping at the time I bought them, about NT$22,000 for the pair.

For context, published llama.cpp numbers for dual-2080-Ti setups already floating around online sit in the 38-50 tok/s band. Those runs are on Qwen3.6, not Qwen3.8, a different model version, and not directly comparable to either number above.

-sm row won't start on any current build: it's not your hardware

If you've read older llama.cpp multi-GPU guides, you've seen -sm row. Don't use it. Its CUDA implementation was removed by PR #24216, "CUDA: remove -sm row, refactor cuBLAS", merged 2026-07-06. On a current build, passing it fails at load with device CUDA0 does not support split buffers.

Check your own binary before spending an evening on this:

strings <path-to-llama-server> | grep -c split_buffer

Zero means your build has row split removed. My old b9870 binary, from before the removal, returned 45.

No --fit under -sm tensor: set your context by hand

-sm tensor is the successor split mode, from PR #19378, "ggml: backend-agnostic tensor parallelism (experimental)", merged 2026-04-09. It's still marked experimental in docs/multi-gpu.md, and one gap in that experimental status will bite immediately: it doesn't support --fit, the flag that otherwise auto-sizes context to whatever VRAM is available.

Leave out -c under -sm tensor and you don't get an error. You get llama.cpp's default context, silently, whatever that happens to be on your build. Always pass -c explicitly.

NCCL is the Linux default, and it's the one that didn't work for me

On Linux, llama.cpp's default allreduce path under tensor parallel goes through NCCL. Mine built and started fine on that default, then failed the moment it had to actually run a reduction between the two cards during a real request.

Setting GGML_CUDA_ALLREDUCE=internal switches to a plain PCIe path using pinned host memory, and that's the path that works, on this hardware, tonight. Why the default one doesn't is where the next section picks up.

Deep dive

Skipping this section costs nothing. Everything above is enough to reproduce the setup. What follows is the order I actually found it in, including the parts where I was wrong.

-sm row dying wasn't a hardware verdict

-sm row refused to load with device CUDA0 does not support split buffers. My first read of that message was architectural: Turing is five years old, and "device does not support X" reads like a compute-capability wall, a feature fenced off from older silicon as newer generations added better primitives.

I went looking for confirmation before doing anything else, and found PR #24216 instead. The row-split CUDA implementation was deleted outright on 2026-07-06, for every GPU generation, as part of a cuBLAS refactor that removed the split-buffer code path entirely. To check, I rebuilt against b9870, the last release before that commit landed, and pointed the exact same two 2080 Tis at -sm row. It started. It ran at 38.4308 tok/s.

The GPUs never lost anything. The feature was removed from the codebase on a fixed date, independent of what card anyone owns. "Device does not support" was describing the binary, not the silicon.

Linking is not the same as working

NCCL is llama.cpp's default allreduce path on Linux under -sm tensor, and my build linked against it cleanly: no missing symbols, no loader errors, and the server came up. I read that as done: symbols resolved, library present, server up. I expected the default path to just work.

It didn't. The server started, answered non-generation requests fine, and then failed the moment a real request forced an actual reduction between the two cards. Switching the allreduce path with GGML_CUDA_ALLREDUCE=internal was the fix. It moves the reduction through pinned host memory over PCIe. I never found out why NCCL threw that error, and I want to be straight about that: I had decided in advance that this branch got one attempt, so when it blew up I took the other path instead of debugging it. Whether NCCL would be faster once fixed is still an open question here, not a settled one.

Linking only proves the dynamic loader found the right symbols at the right addresses. It says nothing about whether the call path those symbols point to is healthy once you actually invoke it. I had conflated "the library loaded" with "the path works." The two turned out to be unrelated facts.

The draft depth that lost on one card won on two

Part 13 swept MTP draft depth on a single card, and n=4 won there, beating n=2 and n=5. I carried that number into the tensor-parallel setup expecting it to still lead, or at worst tie.

I swept again anyway: n=2 landed at 52.50 tok/s (acceptance 0.679), n=3 at 59.4415 (acceptance 78.87% and 74.12% across two samples), n=4 at 57.7197, n=5 at 51.4095, and n=7 at 44.3623, falling further to 37.1107 on a second run. High depth got both slower and less consistent. n=6 never actually ran; a conditional branch in my sweep script didn't fire for it, so the honest answer there is untested, not "worse."

n=3 won, not n=4. Splitting the model across two cards changes how much wall-clock time the big model's own forward pass costs per step, which changes the ratio between draft overhead and verification payoff the whole optimization is balancing. The old winner became the runner-up. Whatever depth won on your last config, re-run the sweep after changing the split mode. Don't carry the number over.

KV cache quantization taxes acceptance, not prefill

I assumed quantizing the main KV cache, the running memory of keys and values for tokens already processed, would cost about what it costs on a single card: a small, roughly fixed hit to decode speed, worth it for the VRAM back. I didn't expect it to interact with MTP at all. The draft head and the KV cache felt like separate concerns.

I ran the same prompt at 64K context, draft depth 3, across three KV types, two samples each: q4_0 gave 57.16 tok/s (acceptance 0.729) shallow and 45.01 (0.579) deep; q8_0 gave 52.67 (0.655) and 51.20 (0.719); f16 gave 59.96 (0.807) and 53.73 (0.726). Prefill speed was effectively identical across all three: 767.60, 760.20, and 766.21 tok/s.

f16 wins on both speed and acceptance, and the flat prefill numbers rule out the obvious explanation. The cost isn't in reading the prompt faster or slower. MTP's guesses get checked against the KV cache's actual stored representation, and a coarser representation gives the draft head less to agree with, so the tax lands on acceptance rate, which then drags decode speed with it, not on the read-heavy prefill phase at all. All three KV types load fine up to the full 262,144 context; f16 at that length costs 17,670 MiB per card. Two samples per type is thin, so treat the ranking as solid and the exact gaps between the numbers as approximate.

Q8_0 weights: the tax lands on code, and f16 KV hits a wall at scale

I expected switching the model weights from Q4_K to Q8_0 to be roughly uniform: better numbers everywhere, a flat cost in speed and VRAM, independent of what kind of text was being generated.

It wasn't uniform. Against the Q4_K production numbers (59.632 code, 34.4 prose), Q8_0 weights gave 55.13 tok/s on code, a 7.5% drop, and 34.90 on prose, a 1.5% bump that is inside noise. The deep-context run came in at 46.10, and VRAM peaked at 19,824 and 18,674 MiB per card. The 8-bit tax landed almost entirely on code generation and left prose alone.

Then I tried pushing Q8_0 weights out to the full 262K context the way I had with Q4_K. With the main KV cache also at q8_0, 131K context gave 54.34 tok/s on code, 31.81 on prose, 42.51 deep. At the full 262,144 it gave 49.93 on a needle-in-a-haystack retrieval check, 48.77 deep, 51.11 code, VRAM peaking at 20,426 MiB with 2,102 MiB of headroom, and the retrieval check passing 3 of 3 across all three workloads.

Then I tried the config that had won the KV-quantization comparison above: main KV at f16, on top of Q8_0 weights, at 262,144. Straight CUDA out-of-memory. A mixed attempt, K cache at f16 and V cache at q8_0, hit a tensor-split assertion failure at the same context length. Backing off to 196,608 finally passed, at 52.32 tok/s, VRAM peaking at 20,944 MiB, headroom down to 1,584 MiB.

f16 KV won the earlier comparison at 64K context on Q4_K weights. It isn't a free lunch once the weights themselves get bigger: on Q8_0 weights, it can't reach full context on this card at all. Whether f16 KV is the right choice depends on the context length you actually need, not just on which KV type wins a benchmark at one fixed length.

Three things I was wrong about that day, in order. I read -sm row's load failure as a hardware verdict on five-year-old Turing cards, and wrote off dual-GPU llama.cpp on this box entirely; upstream had deleted that code path a month earlier, and -sm tensor was alive under a different flag the whole time. I also assumed splitting the model across two cards would be the win by itself, which is why the +1.02% from -sm tensor alone was the most deflating number of the day. And I carried the single-card best draft depth into a new split configuration without re-testing it, which turned out to be wrong on every count once the compute path underneath it changed.

---

Also in this series:

• #1 I scored a 22GB-modded 2080 Ti for ~$340 all-in
• #12 Why isn't your 4-bit quant faster on a 2080 Ti?
• #13 Free 15% speedup: one broken chat template, one starving MTP head