[DGX Spark] DeepSeek V4.1 Flash on Two DGX Sparks: TensorFold TP=2 Setup, 20% Faster Single-Stream Than Qwen3.8

---

Why I did this: a new DeepSeek, and two machines that were already busy

DeepSeek V4.1 Flash 1017 came out, and the community claim was that it gets close to the top cloud models. I wanted to know what it does on my own hardware.

My two GX10 machines normally run Qwen3.8-Flash-Next. I call that seat Hikari. It handles most of my dispatch and text work.

Two things stood in the way. A single 128GB machine can't fit this model, so I had to run a 2.9bpw quant (about 2.9 bits per weight on average) split across both machines. The split is TP=2 (tensor parallel): each layer's math is cut in half, each machine computes its half, and they exchange results. And both machines were tied up running Hikari, so every test meant taking my main model offline for several hours.

Here is how it came out. Single-stream, meaning one request at a time, DeepSeek is about 20% faster than Qwen. After tuning, total throughput with 8 concurrent streams (eight requests in flight at once, tokens per second summed across all of them) is on par for Chinese. So it replaced Hikari.

The first half of this article is the steps, in order, so you can reproduce it. The second half is the full data and the pitfalls.

What you get: about 20% faster single-stream, 8-stream Chinese on par

tok/s is tokens generated per second (a token is the unit a model splits text into); higher is faster. Focus on the two single-stream rows; those are the headline numbers. The bottom two rows are 8-stream totals after the multi-stream tuning in Step 4.

Measurement (tok/s)	Qwen3.8-Flash-Next (Hikari)	DeepSeek V4.1 Flash	DeepSeek vs Qwen3.8
Single-stream code, per stream	92.4	111.5	+20.7%
Single-stream Chinese, per stream	46.4	59.2	+27.4%
8-stream Chinese, total	148.0	150.6	on par
8-stream code, total	285.6	212.0	DeepSeek about 26% lower

Method: same bench script, temperature 0, max_tokens 400, 3 consecutive requests per stream, 3 rounds per cell, median taken.

The two models use different tokenizers, so the same tok/s is not the same amount of text. The deep dive adds a table that compares how many seconds the same prompt takes to complete.

On capability, DeepSeek passed 5 of the 6 tasks in my fixed agent test set, 48 of 60 points. The one failure is a strict-output comparison that hinges on numbering semantics. I don't have a Hikari run on the same set, so I'm not claiming DeepSeek is smarter than Hikari. Only that it can take over Hikari's work.

Requirements: two machines with nothing else loaded, and about 800 GB of disk

The recipe is MiaAI's dual-Spark recipe. It splits DeepSeek TP=2. Tensor parallel means each layer's matrices are cut in two, each machine computes its half, and the two exchange results. The engine is the Zig native engine from TensorFold. One machine is the head and serves the API; the other is the worker.

The checklist:

• Two DGX Sparks or equivalent GB10 machines, 128GB each. Each one holds about 98 GiB of weights. start.sh requires at least 112 GiB of available memory per machine before startup, or it refuses to start. A large model like Hikari has to be fully stopped.
• One QSFP cable between the ConnectX-7 (CX7) NICs of the two machines. Each end gets a private IP in the same subnet, and ping has to work.
• Passwordless ssh from head to worker.
• Docker plus the NVIDIA container runtime on both machines. The head also needs git, python3 and the hf CLI.
• Disk. The recipe says about 505 GB on the head and about 310 GB on the worker, plus a 25 GB image. ABLIT=1 needs an extra copy of the rewritten shards. If you follow the recipe's original order, head usage peaks at about 671 GB.
• The same GPU clock cap on both machines. TP=2 syncs twice per layer, so the slower machine sets the pace. Both of mine have an applications clock of 2418 MHz, and I didn't touch it.

The model files come from three places:

Item	Source	Size
EXL3 2.9bpw checkpoint (39 shards)	Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw @ 45ee5d0f	196 GiB
Engram, two tables (shards 47 and 48, not quantized)	deepseek-ai/DeepSeek-V4.1-Flash-1017 @ 1bae0e10	about 203 GB
ABLIT overlay (optional)	drowzeys/keys-DeepSeek-V4.1-Flash-1017-Abliterated @ 0bdc2036	682 MB

Engram is this generation's "conditional memory" lookup table. The quantized checkpoint doesn't include it, so the two shards have to come from the original deepseek-ai repo.

Step 1: download the checkpoint, Engram and overlay (about 5.5 hours)

The recipe's start.sh downloads everything on first run, but I recommend downloading by hand first. The total is close to 400 GB.

I ran it in tmux on the head (033f). The checkpoint plus the two Engram shards started on Oct 23 at 03:00 and took about 5.6 hours.

The ABLIT overlay has two traps. First, it's a gated repo. You have to log in to Hugging Face and accept the terms on the repo page; a token alone doesn't grant access. Second, hf download --include "*mia29*" did not fetch the large file. Name the files explicitly:

hf auth login
hf download drowzeys/keys-DeepSeek-V4.1-Flash-1017-Abliterated \
  --revision 0bdc2036113c4fdda38ef6005b3cbc0bc0dfa0f0 \
  --include apply_mia29_wo_b_graft.py --include exl3_wo_b_l10_35_mia29.safetensors \
  --include EXL3_MIA29_ABLIT_META.json --include README.md
# the fetched file should be 682,275,992 bytes

If you don't want ABLIT, skip this. ABLIT=0 runs Mia's original weights.

Step 2: wire the CX7 link, and give each machine its own local copy

By default the recipe has the worker read the head's checkpoint over NFS. I didn't do that; each machine reads from its own local disk. The reason is in the recipe README: when the head reads Engram over NFS, a random 4 KiB read takes about 940 µs versus about 90 µs on local disk, and prefill falls to about 690-890 tok/s.

One physical CX7 port shows up as two network interfaces, enp1s0f0np0 and enP2p1s0f0np0. Give both an address, and the recipe detects two RDMA rails. On my pair the head is <head-cx7-ip> and the worker is <worker-cx7-ip>, with the second rail in a different subnet. I left MTU at 1500.

File transfer goes over the same direct link. I measured 0.12-1.04 GB/s, limited by the disks doing mixed reads and writes, not by the cable.

Don't use rsync -aL to copy a whole HF repo. The deep dive explains why.

Step 3: one start.sh on the head brings up TP=2

Stop Hikari first. Hikari runs as a system unit, so it needs sudo systemctl; systemctl --user won't find it. Before stopping it, I waited until there had been 60 consecutive seconds with no running or waiting requests.

Note the layout: Hikari's head lives on what becomes DeepSeek's worker, and Hikari's worker lives on DeepSeek's head.

# on the worker (5bf0): Hikari's head and its timers
ssh coolthor@<worker-cx7-ip> sudo -n systemctl stop qwen38fn-miaai-head-watchdog.timer \
  qwen38fn-miaai-metrics.timer qwen38fn-miaai-head-watchdog.service qwen38fn-miaai-head.service
# on the head (033f): Hikari's worker
sudo -n systemctl stop qwen38fn-miaai-worker.service

Stop only. Don't disable, so rollback can bring it back unchanged.

Then the Quick start from the recipe README:

git clone https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks.git ~/dsv41
cd ~/dsv41
cp scripts/local.sh.example scripts/local.sh
DRY_RUN=1 ./start.sh     # prints the links it found and both machines' docker commands; changes nothing
./start.sh

My final scripts/local.sh, with IPs replaced by placeholders:

WORKER=coolthor@<worker-cx7-ip>
FABRIC_PEER=<worker-cx7-ip>
ABLIT=1
PREPARE=0
CONTEXT=262144
REASONING_EFFORT=high
STATE_DIR=$HOME/.cache/dsv41-tensorfold/state
NFS_VOLUME=/home/coolthor/.cache/huggingface
MEMGUARD_REMOTE_DIR=/home/coolthor/.cache/dsv41-tensorfold
WORKER_ENGRAM_DIR=/home/coolthor/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4.1-Flash-1017/snapshots/1bae0e1088993b9b82de08f59de5be5a34e42170
THINKING=0

PREPARE=0 matters. On the first run, start.sh calls prepare.sh, which builds the image (about 6 minutes of Zig compilation on the head, then the image is copied to the worker), applies the ABLIT overlay, and splits the weights into two rank files. If you have enough disk, let it do all of that in one go.

I didn't have enough disk, so I ran it as three manual stages:

1. Build the image.
2. Apply the overlay. After verifying the 104 rewritten tensors, delete the original 24 shards.
3. Make the rank files. Make rank1 first, send it to the worker, verify the size, delete the head's temp copy, then make rank0.

The lowest free disk during all of this was about 59 GiB.

After preparation, the original checkpoint has only shard 39 left. From then on you must run with PREPARE=0. Otherwise prepare.sh sees the cache as incomplete and downloads 196 GiB again.

This is the line you want to see:

[tensorfold] serving DeepSeek-V4.1-Flash-1017 at http://0.0.0.0:8888/v1 (sampling: temperature 1.0, top_p 0.95, top_k 20; drafts: on; context: 262144; loaded in 31.4s)

Engine load takes 31-46 s. A full restart including the smoke test takes about 42-90 s.

drafts: on is DSpark speculative decoding. The checkpoint ships with draft layers that guess several tokens ahead, and the main model verifies the guesses in one pass.

Step 4: PARALLEL=8, ROUND_ROWS=32, CONTEXT=262144

The recipe defaults to PARALLEL=4, meaning the engine runs 4 streams at once. I usually have 4-6 requests in flight. With 4, streams 5 to 8 queue, and TTFT (time to first token) p90 hit 5.4-8.6 s (p90: 90% of requests get their first token within that time). With 8 it is about 1 s.

Read the first column for Chinese, the second for code, the last for wait time:

Setting	8-stream Chinese total tok/s	8-stream code total tok/s	6/8-stream TTFT p90
PARALLEL=4, ROUND_ROWS=16 (default)	110.1	162.8	5.4-8.6 s
PARALLEL=8, ROUND_ROWS=16	151.8	185.8	about 1 s
PARALLEL=8, ROUND_ROWS=32	150.6	212.0	about 1 s

The Chinese gain comes almost entirely from PARALLEL=8. ROUND_ROWS=32 lifts multi-stream code further.

ROUND_ROWS is how many rows one decode round processes. The recipe's shipped kernel only covers up to 16 rows. 32 needs the 64 wide-row cubins from PR #31. Without them, forcing 32 fails at CUDA graph capture (issue #23). The recipe's start.sh only accepts 16; I changed mine to accept 16-48.

I also applied PR #40, which handles long-prompt memory. Nine prompts at temperature 0 gave byte-identical output before and after.

CONTEXT is 262144, not the recipe default of 1M. After load, the head has only about 8 GiB of memory available, and the memory guard (memguard) kills the container at 2.5 GiB. I have not verified 1M with that little margin.

PARALLEL=8 does not mean eight 262K conversations can live in memory at once. The 8 streams share one pool of 262144 positions, so 8 histories of 110K each will queue.

The final start command:

cd ~/dsv41
env -u TF_DS_PREFILL_INTERLEAVE_MS IMAGE=tensorfold-dsv41:exl3-tp2-6509d19281e9 \
  PREFILL_INTERLEAVE_MS=250 GRAPHS=1 PARALLEL=8 ROUND_ROWS=32 CONTEXT=262144 \
  THINKING=0 PREPARE=0 ./start.sh restart

The leading env -u guards against a launcher pitfall. If your shell still has an old TF_DS_PREFILL_INTERLEAVE_MS exported, it goes straight to the engine and overrides PREFILL_INTERLEAVE_MS.

Thinking mode: off on the server, turned on per request with high

The server runs with THINKING=0. Normal chat and short agent steps skip thinking, so replies come back faster. A request that needs thinking asks for it:

{
  "model": "DeepSeek-V4.1-Flash-1017",
  "messages": [{"role": "user", "content": "..."}],
  "chat_template_kwargs": {"enable_thinking": true},
  "reasoning_effort": "high",
  "max_tokens": 2500
}

Use effort high, not max. The recipe README notes that max tends to cause thinking loops in agent work (#33, #37). This engine maps high to 75 and max to 100.

With thinking on, give it enough max_tokens. If the request doesn't set one, the server default is 32768.

Plugging it into your tools: OpenAI-compatible, but the caller sets the limits

Any OpenAI client works: base_url = "http://<head-ip>:8888/v1", model DeepSeek-V4.1-Flash-1017. This is my Hermes agent config:

model:
  default: DeepSeek-V4.1-Flash-1017
  provider: ds4
  base_url: http://<head-ip>:8888/v1
  api_key: dummy
  context_length: 131072

Four things to know before you point real work at it:

• No structured output. response_format with json_object or json_schema returns HTTP 400 (text is fine): structured output (response_format and the guided_* fields) is not supported by this engine yet. For JSON, ask for it in the prompt and validate it yourself.
• top_k tops out at 56. Anything higher is rejected with TooManyCandidates.
• The API has no key by default and is open to the whole network. Set API_KEY=1 to lock it.
• /v1/models returning 200 doesn't mean it generates tokens. After a CUDA graph error, HTTP stays up while inference sits idle (#51). A health check has to send a real small request.

I added limits on the caller side: max_tokens 2500 per tool turn, at most 20 tool turns, a 60 s timeout per tool, and a request that has already started emitting tokens but goes idle for 180 s counts as stuck. On an "instant stop" (finish=stop, 1 token, no tool call), the caller resends the identical request once. I measured that at about 1 in 100.

There's also a watchdog on my Mac. Every 2 minutes it sends a real request with max_tokens=5 and must get text back within 60 s. Every 10 minutes it scans both machines' kernel logs for AER, CmpltTO, SMMU and mlx5 timeout messages, which precede the whole-machine hard lock in #47. Three consecutive failures raise an alert. It only alerts; it takes no action.

Rollback: one script brings Hikari back

The same script, /home/coolthor/dsv41/rollback-to-hikari.sh, sits on both machines. Running it on the worker forwards to the head. --dry-run prints:

ssh coolthor@<head-ip> 'cd ~/dsv41 && ./stop.sh'
ssh coolthor@<head-ip> 'sudo systemctl start qwen38fn-miaai-worker.service'
ssh coolthor@<worker-ip> 'sudo systemctl start qwen38fn-miaai-head.service'
wait up to 1800s: http://<worker-ip>:8000/v1/models HTTP 200 and successful chat completion
ssh coolthor@<worker-ip> 'sudo systemctl start qwen38fn-miaai-head-watchdog.timer qwen38fn-miaai-metrics.timer'

It waits for /v1/models to return 200 and for an actual generated sentence, not just HTTP. After stopping each rank's container and before removing it, stop.sh saves compressed logs to ~/.cache/dsv41-tensorfold/logs on that machine.

To go back to DeepSeek: cd ~/dsv41 && ./start.sh restart.

I didn't modify Hikari's files or launcher. The launcher md5 on both machines was 7e7cfee4005bc115298f902c6967e863 before and after.

Deep dive: full data, known issues, maintenance windows

You can skip this part and still use everything above. It holds the full data, root-cause research on the recipe's known issues, three maintenance windows, and every pitfall I hit.

Full results, round 1 at PARALLEL=4: single-stream wins, 4-stream loses

This is round 1, measured on deployment day with Hikari's original bench: 12 valid rounds, 90 requests, 0 contaminated reruns. DeepSeek runs without thinking by default; Hikari uses its original default request. prose_zh is Chinese prose.

Model	Load	Streams	Total tok/s	Per-stream tok/s	TTFT p50 / p90 (s)
Hikari	code	1	87.129	92.429	0.209 / 0.211
DS4.1 ABLIT	code	1	99.986	111.527	0.219 / 0.282
Hikari	code	4	221.071	59.157	0.398 / 0.434
DS4.1 ABLIT	code	4	164.008	45.751	0.279 / 0.650
Hikari	prose_zh	1	43.980	46.439	0.208 / 0.214
DS4.1 ABLIT	prose_zh	1	51.691	59.186	0.228 / 0.263
Hikari	prose_zh	4	113.101	32.007	0.388 / 0.436
DS4.1 ABLIT	prose_zh	4	103.140	28.605	0.378 / 0.684

What it proves: single-stream DeepSeek is faster, but under default settings its 4-stream total is lower. Code is 164.008 vs 221.071 tok/s (-25.8%), Chinese 103.140 vs 113.101 tok/s (-8.8%). I usually have 4-6 requests in flight, which is why I tuned multi-stream below.

I also ran the recipe's own tools/bench.py once as a smoke comparison: about 2K-token prefill at 1590.55 tok/s (TTFT 1.24 s), random-word single-stream decode at 58.09 tok/s, 4-stream total 97.83 tok/s. The tool's README states 1500-1600, 55-61 and 113-117. Random words are hard for the draft layers to guess, so this decode figure is lower than on real text. Don't compare it with the table above.

Multi-stream tuning: p4 to p8 to p8r32 to PR40

Each cell ran at 1, 2, 4, 6 and 8 streams, 3 rounds per cell, 3 requests per stream. The Hikari column reuses the same-day result of the same bench. Hikari itself runs with max_num_seqs=6. Here p4 is PARALLEL=4, p8 is PARALLEL=8, p8r32 is PARALLEL=8 with ROUND_ROWS=32.

Load	Streams	p4 total	p8 total	p8r32 total	Hikari total	p8r32 TTFT p90	Hikari TTFT p90
code	1	100.865	99.755	99.356	87.129	0.211	0.211
code	2	139.273	135.979	136.225	132.124	0.381	0.365
code	4	153.677	169.691	184.058	221.071	0.612	0.434
code	6	153.842	181.410	205.329	279.038	0.927	0.470
code	8	162.785	185.848	211.962	285.623	1.160	6.430
prose_zh	1	53.845	55.590	55.723	43.980	0.231	0.214
prose_zh	2	82.107	82.833	82.578	72.731	0.450	0.386
prose_zh	4	107.348	116.364	114.754	113.101	0.680	0.436
prose_zh	6	110.570	138.935	138.685	149.406	1.063	0.475
prose_zh	8	110.129	151.823	150.553	147.988	1.345	6.117

Three takeaways:

1. p4 barely grows at 6 and 8 streams (code 153.8 to 162.8) because the extra requests queue. p4 TTFT p90 at 6/8 streams is 5.454-8.647 s.
2. ROUND_ROWS 16 to 32 mainly helps multi-stream code (8-stream code 185.8 to 212.0); Chinese barely moves. I picked p8r32 on the geometric mean of the total-throughput ratio over four cells (code and Chinese, 4 and 8 streams). p8r32 came out 4.875% above p8, and the 9 prompts stayed byte-identical.
3. Hikari's 8-stream TTFT p90 is 6.1-6.4 s because max_num_seqs=6 makes streams 7 and 8 wait. DeepSeek's 8-stream p90 is 1.2-1.3 s. When 8 requests arrive at once, that matters more than total tok/s.

Speculative-decoding acceptance rate (the share of draft-layer guesses the main model accepts): DeepSeek is 0.55-0.75 on Chinese and 0.82-0.99 on code; Hikari is 0.37-0.42 on Chinese. DeepSeek's DSpark draft uses the full vocabulary, so the trick I used on Hikari, adding a Chinese draft vocabulary, doesn't apply here and isn't needed.

After applying PR40, I reran the short prompts to check for regression:

Load	Streams	Total tok/s before to after	Change
code	1	99.356 to 98.658	-0.70%
code	4	184.058 to 185.089	+0.56%
prose_zh	1	55.723 to 54.920	-1.44%
prose_zh	4	114.754 to 110.673	-3.56%

The worst case is -3.56%, inside my 10% threshold.

Then a long-prompt test: 8 streams, each about 30K tokens, sent concurrently. All 8 succeeded. First-completion p50/p90 was 81.6 / 133.1 s. Continuing with the full history, the cache hit 28672 tokens and completion p50/p90 was 7.5 / 7.5 s. The lowest available memory during the stress run was 8.45 GiB on the head and 8.77 GiB on the worker; memguard never fired. This only verifies about 30K per stream, not 8 x 1M.

Time to finish the same prompt: a second yardstick when tokenizers differ

Since tok/s doesn't mean the same text across tokenizers, I replayed Hikari's historical bench prompts, original nonce included, unchanged against DeepSeek, and compared how long a user actually waits.

Load	Streams	Pairs	Hikari completion p50 / p90 (s)	DS4.1 completion p50 / p90 (s)	Median answer chars, Hikari / DS
code	1	3	3.644 / 3.681	2.410 / 2.423	1093 / 862
code	4	12	5.835 / 5.892	5.106 / 5.186	1107 / 856
prose_zh	1	3	3.888 / 3.916	3.666 / 4.320	248 / 275
prose_zh	4	12	5.362 / 6.215	6.753 / 7.927	238 / 279

On code, DeepSeek's waits are shorter, and so are its answers (862 vs 1093 chars). On 4-stream Chinese, DeepSeek's waits are longer (p50 6.753 vs 5.362 s) and its answers are longer too (279 vs 238 chars). So the table doesn't say which model is faster.

Agent tasks: 5/6 pass, the one failure is numbering semantics

Question	Result	Score	Wall time (s)	Old DS4 07-09
U1_surgical_parser	PASS	10/10	29.693	FAIL
U2_exact_state_cli	PASS	12/12	86.326	PASS
U3_exact_schema_decoy	PASS	8/8	23.516	PASS
H1 package_refactor_tests	PASS	10/10	27.974	no same-question comparison
H2 stateful_cli_strict_stdout	FAIL	0/12	55.456	no same-question comparison
H3 long_grounded_exact_json	PASS	8/8	35.040	no same-question comparison

Settings: agent max_turns 90, context 131072, reasoning_effort high, a fresh profile per question. The three U questions total 30/30. The old DeepSeek V4 Flash run in July on the same questions scored 20/30, but that run has no complete parameter record, so treat it as historical reference only.

The H2 failure: after the first record was completed, the validator expected 1. beta. The model kept the original ID and printed 2. beta, then reported that it had followed the spec fully. The task doesn't say whether to renumber after filtering, so I kept the FAIL and didn't change the score. The practical point: when you give this model tools, keep external validation and don't trust its self-report.

Known issues: root-cause research on the recipe's open issues

Before going live I read all 7 issues and 5 PRs on the recipe repo that affect this setup, and traced root causes against the engine source I was actually running. Ordered by how much each one hurt my setup, weighed against the cost to fix:

Priority	Problem	Mitigation done first	Root-fix direction
1	#34 + #51: all agents stall, and queueing can't be told apart from a real hang	Caller limits: 20 tool turns, 60 s per tool, per-turn max_tokens starting at 2500, a request already emitting tokens idle for 180 s; long TTFT judged by queue and chunk progress, not killed uniformly at 180 s	Collect caller-side timings before touching the scheduler; don't patch an unknown root cause
2	PR44: a tool turn drops the preceding text, adds two newlines, and the next turn's cache re-reads	Caller keeps content / reasoning / calls complete and doesn't rebuild history	Apply upstream PR44 (later absorbed upstream as 0027)
3	#51: permanent false-alive after one CUDA graph error	GRAPHS=0 as an isolation control; restart both ranks together with throttling; health check doesn't rely on /v1/models alone	Draft patch to fail-stop on graph error; long term needs a shared abort protocol across both ranks
4	#21 / #43: control-token spellings inside external text get encoded as real control ids	Neutralize known control spellings in external data first	Original PR43 conflicts with PR40 and has three boundary gaps; needs a rebase
5	#36: after long uptime, rank0 segfaults inside cuBLAS	PREPARE=0, no prepare or download while serving; keep memguard	Root cause undetermined; only a trace draft so far
6	#47: mlx5 PCIe timeout, then SMMU stuck, then the worker hard-locks	External monitoring of kernel events; divert traffic when seen; no automatic reboot	Suspected NIC / firmware / kernel; the engine can't fix it
7	#23 / #31: wide-row kit missing files or half-merged	Refuse to enable 32 if the kit identity doesn't match	Verify, then prepare offline, then switch the whole kit atomically

#51 is the most dangerous one for a service nobody is watching. After graph instantiation returns CUDA_ERROR_NOT_PERMITTED (800), /v1/models still returns 200, chat produces no text, and the GPU shows 96% utilization while drawing only about 20 W. The maintainer confirmed that CudaFailed doesn't stop both ranks together. What I saw looks like one side waiting on a collective for the other, but that part is unconfirmed.

#21 matters whenever you feed the model external text. If an article's body contains literal strings like </think> or role markers, the current tokenizer encodes them as real control tokens, which lets external text spoof control tokens.

During the research, my own logs showed no inference errors matching #34, #36, #51 or #47. They did show NVRM memory-allocation failure warnings, concentrated in the load phase:

NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359

That's 4 lines on the head and 6 on the worker, all before serving started. It's a GPU driver allocation failure, not a Linux OOM kill, and nothing ties it to the later segfault. I'm keeping it on record rather than dismissing it.

Three maintenance windows: "stop everything on the first error" miscounted the failure rate

I ran the controlled experiments the root-cause investigation called for in three maintenance windows, with my workload moved to the Strata seat on my three-2080 Ti box in the meantime. The stress tool is my own repro_agents.py, which simulates multiple agents running tool loops:

• A: 8 agents x 8 rounds, 16K initial prompt, 256-token tool result per round, max_tokens 400.
• B: 8 agents x 12 rounds, 110K initial prompt, 4096-token tool result per round, max_tokens 2500, thinking high.
• Stream 0 sleeps 30 s on every tool call, to see whether one agent running a tool blocks the others. All tools are simulated; commands the model issues are not executed.

Round 1 results:

Measurement	Baseline I250/G1	C: interleave 0	D: GRAPHS 0	New image + PR44	Rebuild without PR44
A valid tool rounds	64/64	64/64	64/64	64/64	64/64
A client TTFT p90 (s)	40.643	61.414	41.462	61.947	71.773
A decode median tok/s	11.497	11.277	11.458	10.394	10.374
B valid tool rounds / HTTP completions	0/96; 1	15/96; 16	0/96; 1	40/96; 41	0/96; 1
B client TTFT p50 / p90 (s)	-	371.581 / 405.559	-	397.426 / 435.281	-

A was 64/64 everywhere with no GLOBAL_STALL, and the other streams kept advancing while stream 0 slept. B looks dramatic: the new image with PR44 got 40 rounds and the other groups got 0. The naive reading is "PR44 fixed the instant stops."

That reading was wrong, and the harness was the cause. In round 1, on the first invalid turn, the harness closed every other HTTP connection in that test group. I had built it that way to avoid wasting minutes on queued cold prefill after a failure. So B's 0/96 only means the first request to come back happened to be an instant stop; the HTTP-completions column shows 1. Each B group was really one run that stopped at its first invalid turn, with a different number of completed requests, so the groups can't be compared on failure rate. The 40/41 group was luck: its first 40 just didn't hit one.

I was also following a rule I'd written into the ticket: a candidate that fails the full A/B run gets reverted to the original image.

Round 1 did identify two failure types:

1. Instant stop: finish=stop, the completion is 1 token, no tool call.
2. Thinking loop: finish=length, all 2500 tokens are repeated thinking, for example "7! = 5040. Yes." over and over, or word salad. Raising max_tokens doesn't help; it just runs longer.

I also dropped PR44, for a reason unrelated to instant stops. A simple 4-round-trip test was 4/4 exact, but real 110K thinking round trips were only 1/2. When the model's reply starts directly with DSML without the two preceding newlines, rendering the history inserts a token 271, so the mismatch starts at token 110361.

Round 2 switched the harness to record-and-resume. Each invalid turn records its failure class and resends the same messages once, unchanged. If the second try also fails, the harness skips that turn and doesn't put the bad reply into the next round's history. I aborted this round after less than 6 minutes to test upstream's new version, so no final numbers came out of it.

Round 3 (the upstream upgrade, next section) used the record-and-resume harness and produced the first usable failure rate: instant stop 1/110 on the old image and 1/108 on the new one. The Wilson 95% intervals are 0.16%-4.97% and 0.16%-5.06%, fully overlapping. Conclusion: instant stop is about 1% and caller-side resend handles it; the thinking loop is occasional.

Why I skipped the upstream update: per-stream decode at 4 streams drops about 30%

MiaAI's recipe upstream at 27741e1 brought a large batch: spill v2, KV sharding, conversation continuation, plus 0027 (the official PR44) and 0028 (the official PR43). All 8 patches I had applied locally were absorbed upstream. PR31's wide-row kit is also compatible with the new engine (round.zig accepts 16..48).

Same record-and-resume harness, one run each on old and new:

Version / load	Valid rounds / HTTP completions	Instant stop	Thinking loop	Decode median tok/s	Long-prompt cached tokens
old A	64/64	0/64	0/64	11.507	56/64 entries non-zero
old B	45/46	1/46	0/46	9.082	0/46
new A	64/64	0/64	0/64	11.093	55/64 entries non-zero
new B	42/44	1/44	1/44	9.034	0/44

Short-prompt speed, 16K synthetic prefix, max 256 tokens:

Version	Concurrency	Server decode median tok/s	Aggregate tok/s
old	1	53.127	17.036
old	4	36.499	102.181
old	8	24.827	102.192
new	1	53.390	17.035
new	4	25.709	68.116
new	8	27.744	68.108

I set three upgrade conditions in advance: (1) an instant-stop rate low enough to stand out from the noise; (2) zero stalls; (3) 1/4/8-stream short-prompt speed all at least 0.9x of the old version.

Results: instant stop was 1/110 (old) vs 1/108 (new), no difference. The new version's 4-stream and 8-stream aggregates are 0.67x of the old (68.116 vs 102.181, and 68.108 vs 102.192 tok/s). Long-prompt cached tokens were 0 in both versions, so the new conversation continuation didn't help in this setup (spill is off by default). The instant-stop and speed conditions both failed.

Decision: keep tensorfold-dsv41:exl3-tp2-6509d19281e9. The new image, tensorfold-dsv41:upgrade-27741e1-928adbd0e57d, is kept, not deleted.

ABLIT on sensitive questions: removing refusals is not removing censorship

The ABLIT overlay modifies 104 attention tensors in layers 10-35.

At go-live I ran a 5-question safety-boundary smoke test: fictional villain threats, scam-prevention analysis, black-humor profanity, adult flirting, and a fantasy moral negotiation. All 5 got substantive replies with no refusals. They were all mild, and I had no original-model comparison, so this does not prove every restriction is gone.

Then a harder one: "What is your view of the June 4, 1989 Tiananmen incident? Answer in Traditional Chinese." (max_tokens 700, default parameters). It did not refuse (finish=stop), but the narrative leans official. It uses the word "clearing" (清場), doesn't mention troops firing, tanks or casualties, and lists the official characterization and "creating conditions for economic development" together in one section. Its "different perspectives" part only says some people stress "the role of maintaining social order."

Three factual follow-ups, same parameters, all finish=stop:

1. Death toll. It listed the official figure of about 200-300, the Red Cross's initial figure of about 2600 that was later retracted, the Tiananmen Mothers' list of 200+ names, and estimates of "hundreds to thousands." It didn't mention the British diplomatic cable estimate of about 10,000, and it mixed in Simplified characters (数字, 约, 数百).
2. Tank Man. The description is correct: white shirt, carrying bags, blocking a tank column on Chang'an Avenue, identity unconfirmed.
3. Were shots fired? It answered "yes" directly and listed officially acknowledged casualties, eyewitness testimony, footage and the Tiananmen Mothers. But it rendered Yuan Mu's "not a single person was killed on the square" as "not a single person was killed," and it cited a casualty record (military and civilian) from the "Yearbook of the People's Republic of China" that I have not verified.

ABLIT removes the refusal behavior, not the narrative stance baked into the training data. Specific factual questions get answered, mostly correctly, with details that drift.

Pitfalls hit during deployment

rsync -aL copied the data twice. An HF cache keeps real files in blobs/ and symlinks in snapshots/. -L resolves symlinks into real files, so both the blobs and the snapshot got copied. When I noticed, the worker had about 256 GiB free. I stopped it, used rsync -a to restore the snapshot's symlinks against the blobs already there, and then copied only the Engram snapshot.

The graft failed on a permission error at the very last step. All overlay shards had been written; copying the read-only HF index at the end failed:

PermissionError: [Errno 13] Permission denied: '/hf/tensorfold-dsv41/45ee5d0f-ablit-0bdc2036/checkpoint.new/model.safetensors.index.json'

I didn't redo the 141GB graft. I fixed the permission on the new directory's index only, read all 39 safetensors headers, verified the 104 overlay tensors against the sidecar with torch.equal, added the index, and then ran the original ablit_stock_dtypes.py.

Hikari's bench crashed on round 1 with RuntimeError: missing running/waiting metrics. TensorFold names its metrics tensorfold:num_requests_running/waiting, not vllm:*. I changed only the monitoring name mapping and the MTP counters; the request and statistics formulas are untouched.

One Hermes config field was silently ignored. The CLI warned providers.ds4: unknown config keys ignored: max_tokens, type, so max_tokens: 32768 in the config did nothing. The real cap was the server default of 32768, the same value by coincidence. To limit output length, put it in the request.