177B on 44 GB of VRAM: Two 2080 Tis Run Qwen3.8-Flash-Next at 98.7 tok/s with Strata

Why I tried two cards: a smarter model for the family server

Part 23 put the 177B Qwen3.8-Flash-Next on a single modded 2080 Ti with Strata. It ran code at 70.3 tok/s on the old build.

That left two questions. How fast do two cards go? And could Flash-Next take over GPU0 and GPU1 from Qwen3.8-27B? The 27B runs the web chat my family uses. It's fast, but it's a 27B, and Flash-Next is the smarter model. I didn't know whether Flash-Next would hold up with several people using it at once.

The box is the same EPYC machine from part 22 and part 17: 128 GB of RAM and three modded 2080 Ti 22GB cards.

Moving Strata from one 2080 Ti to two: the config diff

I cloned a newer Strata (61b3fb5d) into a separate directory, /mnt/nvme1t/strata-upstream, and built it for CUDA 12 and sm_75. The model, the pack and the MTP draft layers are the same files part 23 downloaded.

Two things change in the config compared to one card:

• "gpu": [0, 1]. The single-card setup pinned GPU2 with CUDA_VISIBLE_DEVICES set to its UUID. Drop that for two cards.
• "layer_split": "auto". That's the default. With two or more GPUs, the server adds --layer-split on its own. On my box it split at layer 24.

Here's the full config I used, dual-solo.json. Point --native and --ple-gguf at wherever setup put the two GGUF shards:

{
  "args": [
    "--pack", "/mnt/nvme1t/strata/data/packs/iq3_s",
    "--native", "<model-dir>/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
    "--ple-gguf", "<model-dir>/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
    "--expert-profile", "/mnt/nvme1t/strata/src/data/expert-profile.bin",
    "--expert-cache", "auto",
    "--prefill", "auto",
    "--spec", "4",
    "--spec-min-p", "0.5",
    "--mtp", "/mnt/nvme1t/strata/data/mtp/rt",
    "--max-context", "32768",
    "--kv", "int8",
    "--ple-io", "ram"
  ],
  "gpu": [0, 1],
  "layer_split": "auto",
  "env": { "STRATA_BF16_TC": "1" },
  "parallel": 1
}

Launch it:

.venv/bin/python serve/server.py --engine strata \
  --config /mnt/nvme1t/strata-upstream/dual-solo.json \
  --host 127.0.0.1 --port 8097 --gpu 0,1

The --gpu 0 trap

My first run had --gpu 0 left over from the single-card command. The command line overrides the config, so the engine ran on GPU0 only. It still loaded and still answered requests. It measured 76.4 and 81.8 tok/s, which looked like "two cards didn't help."

Check before you benchmark:

nvidia-smi --query-gpu=index,memory.used --format=csv,noheader

A real two-card run shows both cards at about 21.6–21.7 GiB. The bad run showed 21616 MiB on one card and 4 MiB on the other.

One user: 98.7 tok/s on code, about 15% faster than one card

Method: real OpenAI-compatible chat requests, streaming, reasoning_effort set to none, 3 runs per prompt with different nonces, median. Max 650 output tokens (250 for the 16K prompt). The code answer gets cut off at the limit, the same way on both sides. Speed is Strata's own generation speed. Both sides used 32K context, int8 KV, --spec 4, --spec-min-p 0.5 and STRATA_BF16_TC=1. The one-card column is the same Strata build on GPU2, without --kv-resident.

The first row is the one that matters:

	one card	two cards	change
code / Chinese tok/s	85.6 / 57.0	98.7 / 64.9	+15% / +14%
English tok/s	not measured	62.6
16K prompt: time to first token	not measured	10.4 s
16K prompt: prompt reading	not measured	1,545 tok/s

Each card uses about 21.6 GiB of VRAM.

Against Shio, the production single-card setup from part 23 (128K context, --kv-resident 32768, --ple-io direct), the gap is smaller. Shio does 89.2 on code and 57.8 on Chinese, so two cards are only about 11% faster than that.

Why a second GPU adds only 15%: layer split is a relay, not tensor parallel

Strata spreads a model across GPUs with layer split. Flash-Next has 48 layers. GPU0 gets layers 1–24. GPU1 gets layers 25–48, the output head and the MTP draft layer (a small built-in layer that guesses the next few tokens so the main model can check them in one pass).

For every token, GPU0 finishes its half, hands about 50 KB per position to GPU1, and GPU1 finishes the rest. It's a relay: for one user generating token by token, one card runs while the other waits.

The second card does help a bit. Its extra VRAM holds more hot experts, so the CPU has slightly less to do. But most experts still live in system RAM and the CPU computes them there, the same as on one card.

The other way to use two cards is tensor parallel: both cards compute half of the same layer at the same time. That's how the 27B runs under FastLLM. Strata's CUDA path doesn't have it. The author's answer in issue #156 is that decode is mostly bound by the experts, and dense tensor parallel is not planned. Splitting the dense part across cards wouldn't speed up the CPU's expert work anyway.

Multiple users: 26 to 32 tok/s each with four people

To serve several users at once, set "parallel": 4 and add --batch-mtp.

The table has two columns. Per user is each person's generation speed, not counting the wait for the first token. Combined is all output tokens divided by the time until the slowest request finishes, and it includes the first-token wait. That's why combined is lower than per user at one user.

users	per user tok/s	combined tok/s
1	97.2	88.4
2	47.6	87.7
4	25.9	95.2

Combined throughput barely moves as users are added. Adding users mostly splits the same throughput among more people.

Eight users won't fit. At four, GPU1 is already at 21,996–21,998 MiB out of 22,001.

Turning --batch-mtp off helps at four users, but only on a warm server. After warming up with one and two users first, four users got 116.9 combined and 32.1 each. A cold restart straight into four users got 93.6. So four users land somewhere between 94 and 117 combined, depending on warm-up.

What it feels like when people pile on: one long answer streams at about 90 tok/s. Three more people start asking. The first answer drops to 22–23 tok/s. When they finish, it climbs back to 87–94 within 5–10 seconds.

Strata is plenty for one user; the family server stays on the 27B

For a single user, Strata runs this 177B well, on very little VRAM: two 22 GB cards (one, for the single-card build) plus 128 GB of system RAM for the rest. In part 17, llama.cpp ran the same model across all three cards at 23.14 tok/s with 128K context. Now two cards do 98.7 on code, and the single-card Shio does 89.2 (different quant and context from part 17, so treat it as a rough comparison). Chinese at 64.9 is on par with the 27B's 63.0.

The gap opens up with several people at once. The 27B numbers are from October 5: same GPU0+1, same prompts, FastLLM with tensor parallel across both cards, max_batch=4.

	Flash-Next (two cards)	27B (two cards)
one user, code tok/s	98.7	237.3
four users, combined tok/s	94–117	420
one user, Chinese tok/s	64.9	63.0

With four people, the 27B has about four times the combined throughput. It's also more than twice as fast for one person writing code, but 98.7 is already fast, and on Chinese they're even.

How I'd pick:

• Using it alone and you want the smarter model: Flash-Next on two cards. The 27B has to stop while it runs.
• Family use, several people at once: keep the 27B. Flash-Next gives each person 26–32 tok/s at four users.
• Two separate single-card Flash-Next instances: skip it. They're slower than one two-card instance; details in the deep dive.

So the family server stays on the 27B. Flash-Next stays on the third card as Shio, taking turns with ComfyUI through the switcher from part 23.

---

Deep dive: batch MTP, mixed load, two instances, flags that didn't help, versions

You can skip this section; everything you need to run the setup and make the call is above. This is the data behind it.

Batch MTP on vs off

Each cell is the median per-user speed, then the median combined speed, with the three combined runs in parentheses.

users	--batch-mtp on	--batch-mtp off
1	97.20 / 88.38 (78.04 / 88.46 / 88.38)	94.90 / 86.28 (78.69 / 86.28 / 86.79)
2	47.60 / 87.65 (87.77 / 87.51 / 87.65)	45.25 / 83.91 (83.91 / 84.03 / 83.54)
4	25.94 / 95.18 (96.26 / 94.64 / 95.18)	32.11 / 116.86 (116.86 / 117.10 / 116.40)

Batch MTP wins by 4.5% combined at two users. Off wins by 22.8% at four.

This isn't one flag flipping one thing. With batch MTP off, layer split uses its default pipeline groups: the two stages work on different requests at the same time. With it on, everything runs as one group with batched MTP drafting. Batch groups and batch MTP can't be combined; the source blocks it explicitly.

Batch MTP acceptance, from the engine log: 92.1–93.0% at two users, 92.4–92.9% at four. The batch draft field in /metrics reads 0 because it doesn't record batch drafts, so use the log.

That 116.86 was measured on a warm server. Cold-start checks at four users, batch MTP off:

setup	three runs	median
default groups	90.15 / 93.60 / 97.84	93.60
explicit --batch-groups 2	94.26 / 94.06 / 96.30	94.26

Both logs say "2 groups of 2".

For reference, the old build (6f32ec07) collapsed with parallel: 4: even a single request dropped to 23.9 tok/s, and four users got 90.4 combined. The new build with parallel: 4 and --batch-mtp keeps a single request at 97.2.

Mixed load: what one user sees when others join

The first user's speed every 5 seconds, batch MTP on. Others join at 10, 15 and 20 s and leave around 35, 40 and 45 s. Three rounds.

time	others running	round 1	round 2	round 3
5 s	0	82.0	74.2	82.2
10 s	1	95.0	86.0	91.2
15 s	2	38.6	38.4	37.8
20 s	3	30.4	28.2	29.8
25 s	3	23.2	22.2	22.0
30 s	3	27.4	27.0	27.0
35 s	2	31.0	30.6	29.8
40 s	1	39.8	39.8	41.8
45 s	0	74.2	78.0	73.0
50 s	0	89.0	91.8	87.4
55 s	0	93.8	83.0	92.4
60 s	0	94.4	88.8	93.6
65 s	0	81.0	88.8	91.8

The low point, 22–23 tok/s, is 22–24% of the 98.7 single-user speed.

Two single-card instances instead of one split

The idea: run one Flash-Next on GPU0 and another on GPU1, put LiteLLM in front, and let each card serve its own users. Each instance had parallel: 1, and ComfyUI was running.

users	per user tok/s	combined tok/s (3 runs)	first token
1	82.03	77.99 / 75.00 / 74.67	~2 s
2 (one per instance)	41.45	74.21 / 72.08 / 76.90	~2 s
4	42.38	73.90 / 72.65 / 75.91	half ~2 s, half ~20 s
8	41.27	72.28 / 72.10 / 72.07	four waves at ~2 / 20 / 38 / 56 s

Two users on two separate cards didn't double the combined throughput (75 to 74), and per-user speed halved (82 to 41). So the bottleneck is something the two instances share on the host side. CPU sat around 50%, and NVMe reads were 0.1–0.2 MiB/s. I didn't pin cores or measure memory bandwidth, so "host side" is as far as I can narrow it.

Compared with one two-card layer-split instance, combined throughput is 15.1%, 15.3% and 36.8% lower at 1, 2 and 4 users. The four-user figure is against the warm 116.86; against the cold 93.60 it's 21.0% lower.

Other per-instance settings, combined throughput at 1/2/4/8 users, 3-round medians:

per-instance setting	1	2	4	8
parallel 1	75.00	74.21	73.90	72.10
parallel 2 + batch MTP	59.91	59.63	62.58	61.31
parallel 2, batch MTP off	60.16	60.30	51.37	not run
parallel 4 + batch MTP	57.88	57.11	58.73	not run
parallel 4, batch MTP off	57.93	57.93	48.07	not run

The parallel 2 and parallel 4 runs had ComfyUI stopped to free RAM. Each instance uses about 40 GiB RSS. With parallel 4, available RAM was about 3.9 GiB at idle and dropped to 2.3 GiB under four users, so I didn't run eight.

Cold-start trap. Starting both instances at the same time, neither became ready within 900 s:

EngineDied: the engine did not report READY within 900 s

It wasn't OOM. Each process was at 40–42 GiB RSS with one CPU core pegged, and available RAM bottomed out around 36 GiB. Even when I started them one at a time, the second still got stuck. A perf stack showed cudaHostRegister → pin_user_pages → do_huge_pmd_anonymous_page → compact_zone: the kernel was compacting memory into huge pages while Strata pinned the expert RAM.

The fix was to turn off transparent huge page defrag just for startup:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
# start the instance; mine was ready in 46 s
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

This only affects startup. All the speeds above were measured after switching back to madvise.

--ple-io ram vs direct

The n-gram table is 28.8 GB. direct is Strata's default and reads it from SSD; ram loads it into memory. Two cards, 3-run medians:

metric	ram	direct	change
code tok/s	98.7	97.4	−1.32%
Chinese tok/s	64.9	64.8	−0.15%
16K prompt reading tok/s	1,544.74	1,538.14	−0.43%
16K first token	10.381 s	10.424 s	0.41% slower
4 users, per user tok/s	25.94	26.19	+0.96%
4 users, combined tok/s	95.18	96.15	+1.02%

Everything is within 3%, so Shio switched to direct. Verified on GPU2 afterwards: code 82.9 / 89.2 / 89.7, median 89.2; Chinese 57.8.

Available RAM didn't visibly go up (single-user code run: 69,932 MiB with ram, 69,925 MiB with direct). The table is probably still sitting in the page cache. I'm not claiming direct saves 29 GB.

Flags that didn't help on the old build

These ran on the old build (6f32ec07) on GPU0+1, 3-run medians. The baseline had --remote-expert-opt on.

setup	code	Chinese	English	16K first token
default layer split	92.3	58.9	60.1	12.6 s
+ --ple-io ram	92.0	58.7	59.9	12.6 s
+ STRATA_BF16_TC=1	91.4	60.0	60.0	10.6 s
--peer-device 1	82.3	55.8	54.9	13.1 s
--expert-cache-device1 auto	48.4	38.3	34.7	13.2 s
explicit split at 24 + STRATA_STAGE_TRIM=1	89.1	60.5	59.0	10.5 s

• STRATA_BF16_TC=1 only helps long prompts.
• --peer-device 1 makes GPU1 stop being a stage and instead help GPU0 compute experts over NVLink. It's about 11% slower. GPU0 still runs all the dense work alone and waits at every layer. The source also shows this path stages data through pinned host memory, so NVLink doesn't always mean a direct card-to-card copy.
• --expert-cache-device1 auto turns GPU1 into an expert helper: −47%.
• Three cards (October 4): code 81.1, Chinese 54.5, against 92.1 and 60.1 on two.

What could still make two cards faster

--pipeline-windows 2, not tested. It's for a single user: GPU0 speculatively computes the next window while GPU1 verifies the current one, and rolls back on a miss. It needs MTP plus layer split, can't be combined with --batch, and needs a second verifier buffer (about 160 MiB) plus GDN snapshots on each card. An upstream report on another pair of 2080 Tis (IQ3_XXS) shows +11.1% (issue #1349). Issue #1143 reports it running with no errors or hangs. My cards are nearly full, and the extra VRAM may push experts out of the cache. My plan is an A/B/A run (0 / 2 / 0), adopted only if warm single-user speed goes up by 8% or more.

Full expert parallel only exists for SYCL (Intel Arc), in PR #1798. It's unmerged and has no CUDA version.

Writing tensor parallel myself. That's about 96 cross-card syncs per token (48 layers × 2). Each sync moves little data, so latency dominates. My estimate for a single user is somewhere between −20% and +5%. I haven't implemented it.

Versions and caveats

• Strata 61b3fb5d (0.1.42 dev tree, 2026-10-10) in /mnt/nvme1t/strata-upstream, CUDA 12, sm_75. The old 6f32ec07 (engine 0.1.39) install is untouched.
• The new two-card runs did not use --remote-expert-opt; upstream docs say it's incompatible with batch MTP. The old October 5 runs did. So the jump from 91.4 (old, two cards) to 98.7 (new, two cards) isn't all down to the version.
• 27B: October 5, Hina, FastLLM, model qwen38-27b-ud, tensor parallel over two cards, max_batch=4. The service on port 8082 runs a different model now; these numbers are from October 5.
• Two-card measurements meant stopping Hina: 42 minutes for the layer-split round, 90 minutes for the two-instance round. Both ran with a dead-man timer on ai-lab that restored Hina and ComfyUI automatically.

Not measured yet

• --pipeline-windows 2 (A/B/A).
• A capability comparison of Flash-Next vs the 27B. An October 5 five-task set scored the 27B at 48/60 and Flash-Next at 40/60, but five tasks is too small a sample and it contradicts what I see when dispatching real work, so I'm not drawing a conclusion from it.
• Image input, tool calls and 128K context on two cards with the new build.

---

Also in this series:

• #23 Qwen3.8-Flash-Next 177B on One RTX 2080 Ti: Strata Runs Code at 70 tok/s
• #22 NVFP4 Without FP4 Hardware: Qwen3.8-27B on Two RTX 2080 Tis at 151.7 tok/s
• #17 A 177B MoE on Three Modded 2080 Tis: 23 tok/s at 128K, 77 on a File Edit