[vLLM] Qwen3.8-Flash-Next TP=2 on Two DGX Sparks: 51.9 tok/s

Preface

Two movers carrying a couch each take half the weight, but they still walk the whole staircase and still have to match steps at every turn. That is roughly what a second DGX Spark does for decode: the weight reads split, the per-step overhead does not, and the result is +24.5% rather than +100%.

This is Part 46 of the DGX Spark series. The direct prerequisite is Part 45, which covers the single-node NVFP4 recipe — the checkpoint, the nine bind-mounted overlays, PLE modes, MTP, and where the 41.7 tok/s baseline comes from. I am not repeating any of it here.

This article covers what changes when you add the second machine, and the short version is two things: bigger models fit (242 GiB across the pair against 121 on one) and it is faster. Below: what it costs in disk and memory, how to cable two ConnectX-7 NICs without losing a day to a cable that turns out to be fine, and where the gain actually comes from.

What you get: 242 GiB between them, 51.9 tok/s, a 2.07x KV pool

Look at the first row for speed and the second for capacity. The second row is the one that decides whether a model you could not load before will load now.

measurement	one Spark (TP=1)	two Sparks (TP=2)
40-prompt harness median decode	41.7 tok/s	51.9 tok/s (+24.5%)

⚠️ The two medians are not the same kind: 41.7 is the 40-prompt median of a single TP=1 run, while 51.9 is the median of three TP=2 runs, each itself a 40-prompt median. The three were 56.6, 51.7 and 51.9, a 9.5% spread. | TTFT, short prompt | 0.26 s | 0.17 s | | KV cache pool | 1,027,392 tokens | 2,128,927 tokens (2.07x) |

Same checkpoint, same harness, thinking off, concurrency 1.

I ran the TP=2 harness three times: 56.6, 51.7, 51.9. Median 51.9, spread 9.5%. My first pass computed the speedup off 56.6 and reported +35.7%, which is selective reporting — picking the best of three runs and comparing it to a single baseline. The headline is the median.

A per-prompt probe, same prompt text on both configurations and warmed up, shows where the gain lands. The third row is the one that needs a warning label.

prompt type	one Spark	two Sparks
code	45.25 tok/s	61.79 tok/s
English prose	28.16 tok/s	37.87 tok/s
Traditional Chinese prose	15.24 tok/s	33.16 tok/s

The throughput gain on Chinese prose is the largest in the table, and it is not a TP=2 result. The two configurations also differ in PLE mode, and the MTP draft head reads the PLE table: the single node ran PLE_MODE=staged (table on disk, fetched row by row), the two-node configuration uses PLE_MODE=none. The single-variable test — one node with PLE_MODE=none — has not been run. Until it is, that row is an observation after a configuration change, not a tensor-parallel gain.

Both machines need a full copy: disk x2, memory x2

This is where most people get it wrong, so it goes first. vLLM cross-node tensor parallel does not put half the weights on each node. Sharding happens after load, at the tensor level, which means every node needs a full copy of the model directory.

The second row is the one that surprises people who sized their disks for "half the model each".

requirement	value for this recipe
memory budget	121 GiB MemTotal per node x 2 = 242 GiB, not pooled (OS and CPU share it — measure available separately)
disk	124 GiB (132.7 GB) per node = 265 GB total
container image	22.2 GB, on both machines
driver / kernel	identical on both; mine are 580.173.02 and 6.17.0-1032-nvidia
launcher script	byte-identical on both; verify with sha256sum

Two of those rows have failure modes worth naming. If the worker is missing the container image, it stalls in docker pull for ten-plus minutes and from the head you see nothing except a rank that never joins. And the launcher script has to actually match: I edited only the head, scp silently failed to land the file on the worker, bash -n passed against the stale copy, and the worker exited with LANE must be A or B. A syntax check does not catch a file that never arrived.

Copying the weights over the ConnectX-7 link took 2 min 43 s at 812 MB/s. The bottleneck there is ssh's AES, not the cable — the link was 4% used.

rsync -a --info=progress2 -e "ssh -c aes128-gcm@openssh.com" \
  /home/coolthor/models/qwen38-fn-nvfp4-nvidia/ \
  sleipnir-2:/home/coolthor/models/qwen38-fn-nvfp4-nvidia/

# then, on both nodes:
sha256sum ~/patches/*.sh

Cabling: one port is two interfaces, and RoCEv2 needs the IPv4 GID entry

The official procedure is NVIDIA's connect-two-sparks playbook, and it works. Three things in it are easy to get wrong, and each one produces a symptom that looks like broken hardware.

First, use the interface pair that ibdev2netdev reports as (Up). On port 0 that is enp1s0f0np0 and enP2p1s0f0np0.

Second, those two names are not a typo. One physical port exposes two logical interfaces across two PCIe domains, and both need an IP address to get full bandwidth. That is why the official measurement procedure runs two streams and adds them. Configure one and you get half, then start blaming the cable.

# /etc/netplan/40-cx7.yaml   (chmod 600; the other machine uses .13)
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp1s0f0np0:
      addresses: [192.168.200.12/24]
    enP2p1s0f0np0:
      addresses: [192.168.201.12/24]

Third, the RoCEv2 GID index only exists once IPv4 is configured. On my machine the table then looks like this:

0   link-local   RoCEv1
1   link-local   RoCEv2
2   IPv4         RoCEv1
3   IPv4         RoCEv2   <- RoCEv2 needs this one

⚠️ Index 3 is not universal — the table is built from whichever addresses are actually configured, so add IPv6 and you get more entries. Read your own table and match on address family and type.

Until an IPv4 address is assigned, only entries 0 and 1 exist, and RoCEv2 then fails with Failed to modify QP to RTR and Unable to Connect the HCA's through the link. That message reads like a dead NIC or a dead cable. On my pair it meant IP was not configured yet.

MTU does not need changing. NVIDIA's own example output shows Mtu : 1024[B] (netdev 1500) reaching 92 Gb/s. I had assumed 9000 was the last untouched knob, and it was the wrong direction entirely.

Verify with two streams and add them:

# NVIDIA's procedure is four terminals on two port pairs, running at the same time.
# On the far node, one server per interface:
ib_write_bw -d rocep1s0f0   -p 12000 -D 12 --report_gbits
ib_write_bw -d roceP2p1s0f0 -p 12001 -D 12 --report_gbits
# On this node, both clients concurrently:
ib_write_bw -d rocep1s0f0   -p 12000 -D 12 --report_gbits <peer-ip-1>
ib_write_bw -d roceP2p1s0f0 -p 12001 -D 12 --report_gbits <peer-ip-2>

🔴 The two streams have to overlap. Run them one after another and each will approach its own single-link ceiling, so the sum means nothing.

Mine: 92.57 Gb/s each, 185.14 Gb/s aggregate. That 92.57 is the same number as NVIDIA's documented example output for NIC1, so it is the expected result rather than a good one.

That number has a PCIe explanation, and it is the same two-domain story as above: lspci -vv shows the CX7 at Speed 32GT/s, Width x4 per link, and dmesg prints 126.028 Gb/s available PCIe bandwidth for one of them. No single stream gets past about 126 Gb/s, which is exactly why the aggregate has to come from two. NVIDIA's own documented example is asymmetric for the same reason: 92.57 + 97.28 = 189.85 Gb/s.

Start the worker first, then the head

I used the recipe author's SPEED profile (Kai / Tech2Wild) and changed four environment values: the CX7 addresses, NCCL_IB_ADDR_RANGE (pointed at the CX7 subnet, the 192.168.200.x side here), --served-model-name so the server accepts requests under two model IDs, and MODEL_HOST.

# SPEED profile, unchanged from the recipe
PLE_MODE=none GRAPHS=nocompile MTP=3 SEQS=6 CHUNK=4096 \
  KV=fp8_e4m3 GMU=0.70 MAXLEN=262144

# the paths I changed
MODEL_HOST=/home/coolthor/models/qwen38-fn-nvfp4-nvidia

Order matters. Start the worker first — it runs --headless and exposes no API — then the head. Reversed, the head cannot find the second rank.

🔴 Do not set GRAPHS=compile. Inductor's autotune duplicates the n-gram table, which costs roughly 24 GiB per rank at TP=2; the author's script comment records this starving and rebooting two Sparks on 2026-09-05. With nocompile, mem_avail settles at 25-27 GB.

Verify with a completion, not /health

/health answering proves the head's web server is up and nothing else. Send a real request instead:

curl -s http://<head-ip>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen38-flash-next",
       "messages":[{"role":"user","content":"Say hello in one sentence."}],
       "temperature":0, "max_tokens":64}'

TP=2 cannot serve with one rank missing, so the head answering at all proves both machines are alive. That is a stronger signal than pinging both boxes: my Tailscale path intermittently degrades to 200 ms RTT, and ping keeps succeeding through that.

One log line looks alarming and is not. distributed_init_method=tcp://<ip>:29533 is the rendezvous bootstrap channel, not the data path. Seeing tcp there does not mean NCCL fell back to sockets.

Deep dive: debugging notes

You can stop here if you just want TP=2 running — everything above is the working recipe. What follows is where the time went, which measurements settled which question, and two conclusions I published and had to withdraw.

Why was a working cable only doing 13 Gb/s?

Three signals said the hardware layer was fine. Both ends negotiated 200 Gb/s. ibdev2netdev showed both interfaces (Up). ib_write_lat reported t_typical of 1.99 µs, completely normal. And ib_write_bw gave 13.2-13.9 Gb/s, 6.7% of line rate.

What I expected, and why you might expect the same: I read those three green signals as "the physical layer is healthy, so this is a configuration problem", and spent the afternoon on QP count, message size, memlock and MTU. The hole in that reading is that ethtool's 200000 Mb/s is a negotiated link speed and ib_write_lat's 1.99 µs is a single-operation latency. Neither one measures throughput. None of the three green signals was looking at the thing I cared about.

The cable was the obvious suspect. ethtool -m reports it as an OEM Q112P-400G-0.5M — I could not find a manufacturer page for that part number, so treat it as what the module claims about itself — and it is not on NVIDIA's approved list (approved are Amphenol NJAAKK0006 and Luxshare LMTQF022-SD-R). An unapproved copper DAC barely holding 200G on FEC is a very plausible story, so I checked the FEC counters over 5 seconds of traffic.

counter	delta over 5 s
rx_corrected_bits_phy	+10,214
rx_err_lane_0	+2,209
rx_err_lane_1	+8,005
rx_pcs_symbol_err_phy	0

That last row acquits the cable: every error was corrected by RS-FEC and nothing reached the packet layer. ⚠️ I originally divided by application throughput to get a pre-FEC BER of about 1.5e-7. That denominator is wrong — the counter wants physical bits, and the same deltas against a 200 Gb/s line rate land nearer 1.0e-8. Naming a BER needs the matching physical-bit delta, which I did not capture. What I can say is that every error was corrected and the cable was fine.

It was also insensitive to every knob: 1QP/64KB 13.49, 8QP/64KB 13.25, 1QP/1MB 13.30, 4QP/256KB 13.24, memlock unlimited 13.46. TCP was equally slow (iperf3, 8 streams: 15.9 Gb/s, zero retransmits), so this was not RDMA-specific. Loopback on a single machine ran 11.43 Gb/s — slow with the cable entirely out of the path.

The decisive measurement was the PHY byte counters. Over a 10-second transfer the sender's tx_bytes_phy rose by only 17.86 GB, which is 14.3 Gb/s. The receiver's rx_bytes_phy rose by exactly the same amount, and tx_discards_phy, rx_discards_phy, rx_pause_ctrl_phy and rx_out_of_buffer were all 0. Nothing was being dropped. The NIC simply never sent more.

That pair of counters separates two causes that look identical from outside, because both present as low bandwidth:

• small tx_bytes_phy delta, zero discards → the host is not sending
• large tx_bytes_phy delta, nonzero discards → it is sending and the path is dropping

Root cause, from NVIDIA developer forum threads 363461, 370035 and 373538: after plugging or re-seating a QSFP cable, the system can stay in a degraded state until it is rebooted. One case went from 13.39/13.76 to 98.01/98.01 after a reboot (thread 363461, post 10). ⚠️ The successful report in thread 373538 explicitly describes a full power drain, not just a reboot — do not collapse the two. And the comment that the dmesg line Detected insufficient power on the PCIe slot (27W) is cosmetic came from forum user mashie, whose profile carries no NVIDIA staff flag, so read it as one participant's view rather than an official position.

After my reboot: 92.57 x 2 = 185.14 Gb/s. Re-verified after a later driver upgrade: 92.56 x 2 = 185.12.

Why did netplan generate fail on a file I never wrote?

The official cabling procedure has you write a netplan file. netplan generate then failed outright on both machines:

Invalid YAML: control characters are not allowed

The culprit was not my file. It was /etc/netplan/90-NM-2edf06d6-....yaml: 715 bytes of all-NUL, same filename, same size and the same 2025-09-29 factory date on both machines. That UUID is not in either machine's NetworkManager connection list, so it is an orphan. Moving it out of /etc/netplan/ unblocks everything.

There is a related red herring: netplan get renderer crashes with a UnicodeDecodeError, which looks like netplan itself is broken. It is the CLI failing to decode while printing the parse error above.

The interconnect sits at 0.3% during inference

I measured this with PHY byte counters rather than inferring it from the absence of NCCL warnings. NCCL_DEBUG=WARN only prints when something is wrong, so silence is not evidence.

Over 99 seconds of inference the ConnectX-7 moved 6.87 GB, about 0.555 Gb/s — 0.3% of the 185 Gb/s link. Discards stayed at 0 throughout, and the worker's tx matched the head's rx exactly.

The mechanism: single-stream decode exchanges activations across machines, not weights. Weight reads are local, each node reading its own half; the cross-machine traffic is one hidden-dimension tensor per step. Three orders of magnitude apart.

This also explains a knob that did nothing. NCCL_CHANNELS=8 is pinned to 8 in TJ Klug's two-Spark recipe, and here it made no measurable difference: a link running at 0.3% has nothing for eight channels to carry.

Effective weight bandwidth lands at 1.41x, and the link is not the limit

If decode were purely weight reads and the weights split exactly in half, the speedup should approach 2. It does not. Measured effective weight bandwidth, using MTP-disabled runs as a proxy, came out at 1.41x, and the ratio held to within 0.2% across three prompt types.

The link's throughput is not the explanation: the 0.3% figure above rules that out. It does not rule out collective latency or synchronisation cost — that needs latency profiling I have not done.

Where it might be, in order of how much I would bet on each:

1. fixed per-step overhead — kernel launch, scheduling, the fixed cost of all-reduce
2. the difference in PLE mode and gpu-memory-utilization between the two configurations

To tell those apart I need one more test: one node, PLE_MODE=none, MTP off. I have not run it. So for now the missing 0.59x can only be described as "not the network".

Retracted. I previously wrote up a step-time decomposition — 34 ms weight read plus 20 ms other — as established. It assumed all-reduce was approximately zero, and cross-checking it against 9.27 GiB/token produced an implied per-rank bandwidth that did not hold. Two measurement points cannot solve three unknowns.

Five config variants, all inside my own run-to-run spread

The recipe author's results directory holds 84 JSON files, 69 of them complete 40-prompt harness runs, which means the remaining knobs can be looked up instead of re-measured. TP=2 medians from that corpus, against mine:

configuration	TP=2 median
speed_mtp4_ixs (MTP=4 + index share)	55.2
ram_mirror	53.7
speed_dv64k (reduced-vocabulary drafting)	53.2
speed_async (ASYNC_SCHED=1)	52.4
speed_nccl8 (NCCL_CHANNELS=8)	51.9
my SPEED config	51.9 (single best 56.6)

My own three runs spanned 51.7 to 56.6, a 9.5% spread, so every gap in that table is smaller than my run-to-run variation — and my single best 56.6 is my own local number rather than part of that corpus, whose highest TP=2 concurrency-1 median is 55.2. Nothing in those 69 runs is cleanly faster than what I am already running.

You can trade speed for a much larger KV cache. The CONTEXT profile (PLE_MODE=mmap GRAPHS=piecewise MTP=4 SEQS=8 GMU=0.80) pushes the KV pool to 5.87M tokens at the cost of dropping to 35.8 tok/s.

The 0.046 quality drop is not a TP=2 cost

My harness auto_score went from 0.889 on the single node to 0.843 at TP=2, a 5.2% drop. All three TP=2 reruns returned exactly 0.843, spread 0.000, and I declared the quality cost real on that basis.

That was wrong. Reproducibility only proves the number is deterministic for that configuration. It says nothing about whether the gap between configurations means anything.

Those 69 runs settle it:

set	auto_score range
TP=1 configurations	0.815 - 0.919 (span 0.104)
TP=2 configurations	0.821 - 0.906

The ranges almost entirely overlap. One TP=1 run at 0.815 sits below my TP=2 result, and three TP=2 runs at 0.906 sit above my TP=1 result. A 0.046 gap is smaller than the normal variation between configurations on either side.

Remote operation with no BMC: the rollback path does not exist

These two machines sit in an office. There is no BMC. That makes one thing worth checking before any kernel or driver work, and it failed for me:

modinfo -k <old-kernel> -F version nvidia
# ERROR: Module nvidia not found

After the kernel upgrade /boot still holds the old vmlinuz, so GRUB offers the old kernel — but the module package was removed, so booting it gets you a machine that cannot see its GPU. The rollback path looks like it exists and does not. The only real safety net is touching one machine at a time.

Tailscale degrades rather than disconnecting, which is its own trap. Ping keeps succeeding while RTT goes from 3 ms to 120-200 ms and the SSH handshake times out. That is the signature of falling back to a DERP relay, not a machine fault. To tell them apart, ssh a node at a different location first. To work around it, drive the worker from the head over the ConnectX-7 link.

Last one, still open: TP=2 does not survive a reboot. Both nodes run a manual docker run --restart no, and the systemd unit for the single-node deployment is still enabled, so rebooting the head silently reverts to one machine. Not fixed yet.

Takeaways

Two things here transfer outside this hardware.

When you see low bandwidth, first separate "not sending" from "sending but dropping". PHY byte counters plus discard counters cut cleanly between them, and that split applies to any network bottleneck, not just RDMA. It is what ended the cable investigation, and it would have ended it hours earlier.

A liveness probe should key on a signal that must visibly break. The head answering a completion proves both ranks are present, because TP=2 cannot serve without both; pinging each machine gets fooled by the Tailscale degradation described above.

And the bottom line on the second machine: 121 GiB becomes 242, the KV pool goes 2.07x, decode goes +24.5%, and the link still has 99.7% spare. Both things hold at once: a model that would not fit now fits, and it runs faster. If you expected decode to double, it does not. Capacity, bandwidth and headroom all do.