~/blog/llama-cpp-upgrade-chunked-prefill-port

改裝 2080 Ti 22G · part 30

[Benchmark] Upgrading llama.cpp on an RTX 2080 Ti: Carry Chunked Prefill Forward and Short Requests Finish in 5.4 s

❯ cat --toc

TL;DR

I upgraded llama.cpp from b11588 to b12595 on a modded RTX 2080 Ti 22G. Decode got 3–10% faster and prefill 13% faster. But my hand-ported chunked-prefill patch didn't come with it: upstream ignores my LLAMA_PREFILL_CHUNK variable without a warning, and its own version (PR #10718) is still open. With a 30K-token prompt in flight, short requests took 16.8 s instead of 7.3 s. The upgrade only stays a win if you carry the 35-line patch forward. With it, short requests finish in 5.4 s.

Modded 2080 Ti 22G series #30 cover: a conveyor feeding a long stack of documents into a graphics card in slices, with small glowing cubes slipping in between the slices

Why I upgraded

One modded RTX 2080 Ti 22G runs MiniCPM-V 4.7 35B-A3B for my family's chat server and a news pipeline. I set that up in #25. It ran llama.cpp b11588 plus my own hand port of chunked prefill, switched on by an environment variable called LLAMA_PREFILL_CHUNK. That name is mine. Upstream has never heard of it.

The chunk size of 64 came from a sweep in part 25. I tried 512, 256, 64, and 32. At 64 and 32, short requests came back within a second of each other, and the long request was only 3% slower.

A month later llama.cpp was at b12595. I wanted to know two things: how much faster is it, and do I still need the patch?

A few terms first. A token is the unit the model reads and writes, roughly a word piece, and tok/s is tokens per second. Prefill is the model reading the whole prompt before it writes anything. Decode is generating the answer one token at a time. TTFT (time to first token) is how long you stare at a blank reply before the first word shows up. My server runs 4 parallel slots, so up to 4 requests share the GPU at once.

Decode is 3–10% faster on b12595, prefill 13% faster

I ran the new build without my patch and compared it with the old build. Look at the 4-stream rows: that's where the gain is biggest.

Test (tok/s)b11588 + patchb12595, no patchChange
Chinese, 1 stream102.89105.65+2.7%
Chinese, 4 streams (per stream)48.1252.25+8.6%
Code, 1 stream102.21105.71+3.4%
Code, 4 streams (per stream)47.9752.69+9.8%
8K prompt prefill20802359+13%

Quality didn't move. The new build scored 129/140 on my evaluation set, same as the old one, and it read every receipt in the OCR check correctly.

So far, a free upgrade.

But short requests wait 16.8 s behind a long prompt

Then I ran the test that matches real life at home. Someone pastes a long document. A moment later, someone else asks a short question.

The test sends one 30K-token prompt, then three short prompts 0.15 s later. Look at the "short done" row: that's how long the person with the quick question waits.

Mixed loadb11588 + patchb12595, no patch
Short requests, TTFT1.10 / 1.10 / 1.10 s2.02 / 15.50 / 2.02 s
Short requests, done7.29–7.30 s16.75–16.99 s
Long request, done22.55 s17.99 s

The long request got 4.5 s faster. Every short request waited about 10 s longer. One of them sat for 15.5 s before the first word appeared.

Why: upstream ignores LLAMA_PREFILL_CHUNK, and PR #10718 is still open

The new run still had LLAMA_PREFILL_CHUNK=64 set. Upstream llama.cpp doesn't read that variable. So there's no error and no warning. It just does nothing.

llama-server works in rounds. Each round it hands the GPU one batch of tokens, up to 2048 in my config. Without the patch, a long prompt fills the whole batch by itself every round. A 30K-token prompt takes about 15 rounds, and the short requests get no room in any of them.

Two timelines: without the patch the long prompt prefills as one block and three short requests wait behind it; with the patch the long prompt is cut into slices and short requests decode between them

Upstream does have a chunked-prefill change: PR #10718. It was opened in December 2024. As of November 2026 it's still open.

So upgrading llama.cpp means carrying the patch forward yourself.

The fix: re-apply a 35-line patch to one file

The patch touches one file, tools/server/server-context.cpp, in two spots. The diff is 35 lines.

  1. While other requests are generating, admit at most LLAMA_PREFILL_CHUNK new prompt tokens per iteration.
  2. If several requests are prefilling at once, split that iteration's budget evenly among them. Each one's cap is its share, with a floor of 8 tokens.

After you clone llama.cpp (next step), save this as chunkpp.patch in the llama.cpp folder:

diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
--- a/tools/server/server-context.cpp
+++ b/tools/server/server-context.cpp
@@ -3941,6 +3941,19 @@ private:
         int32_t n_batch  = llama_n_batch(ctx_tgt);
         int32_t n_ubatch = llama_n_ubatch(ctx_tgt);
 
+        // [local port of ggml-org/llama.cpp#10718] chunked prefill: while some slots are generating,
+        // cap the prompt tokens admitted per iteration so long prefills do not stall them.
+        // LLAMA_PREFILL_CHUNK=<tokens> enables it (0/unset = upstream behaviour).
+        static const int32_t prefill_chunk = [] { const char * e = getenv("LLAMA_PREFILL_CHUNK"); return e ? atoi(e) : 0; }();
+        if (prefill_chunk > 0 && batch.size() > 0) {
+            n_batch = std::min<int32_t>(n_batch, (int32_t) batch.size() + prefill_chunk);
+        }
+        int32_t n_slots_pp = 0;
+        for (auto & s_ : slots) {
+            if (s_.state == SLOT_STATE_PROCESSING_PROMPT || s_.state == SLOT_STATE_STARTED) { n_slots_pp++; }
+        }
+        const int32_t n_chunk_pp = (prefill_chunk > 0 && n_slots_pp > 1) ? std::max<int32_t>(n_batch / n_slots_pp, 8) : INT32_MAX;
+
         auto & alora_scale       = batch.alora_scale;
         auto & alora_disabled_id = batch.alora_disabled_id;
 
@@ -4406,7 +4419,9 @@ private:
                     const int32_t n_decision_first = slot.task->type == SERVER_TASK_TYPE_DECISION ? slot.task->decision.pos_first() : -1;
 
                     // add prompt tokens for processing in the current batch
-                    while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
+                    int32_t n_cur_pp = 0;
+                    while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch && n_cur_pp < n_chunk_pp) {
+                        n_cur_pp++;
                         // get next token to process
                         llama_token cur_tok = input_tokens[slot.prompt.n_tokens()];
                         if (cur_tok == LLAMA_TOKEN_NULL) {

If you've never built llama.cpp, you need git, CMake, and the CUDA toolkit. Clone the repo, check out the exact commit I used (e46069c40, build 12595), apply the patch, and build only the server:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout e46069c40
git apply chunkpp.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j 12

75 is the 2080 Ti's CUDA architecture. Use your own card's number. On other commits the line numbers may drift. If git apply fails, paste the two hunks in by hand. They're short.

Then start the server with the variable set. Replace model.gguf with your model file:

LLAMA_PREFILL_CHUNK=64 ./build/bin/llama-server -m model.gguf -np 4 -c 131072 -b 2048 -ub 512 -fa on -ngl 99

Unset the variable, or set it to 0, and you get upstream behavior back.

With the patch: short requests finish in 5.4 s, and 4-stream decode is 11% faster than the old build

Here are all three builds side by side. Look at the "mixed, short done" row first, then the 4-stream rows.

Testb11588 + patchb12595b12595 + patch
Chinese, 1 stream (tok/s)102.89105.65108.48
Chinese, 4 streams (tok/s per stream)48.1252.2553.77
Code, 1 stream (tok/s)102.21105.71107.20
Code, 4 streams (tok/s per stream)47.9752.6953.18
8K prompt prefill (tok/s)208023592390
Mixed, short done7.29–7.30 s16.75–16.99 s5.21–5.41 s
Mixed, short TTFT1.10 s2.02–15.50 s0.70–0.88 s
Mixed, long done22.55 s17.99 s19.53–19.77 s

Against the old build, Chinese 4-stream decode is up 11.7% and code 4-stream is up 10.9%. Single stream is up about 5%. With a long prompt in flight, short requests finish about 2 s sooner than they did on the old build.

The cost is on the long request. It finishes 1.5–1.8 s later than on the unpatched new build. In exchange, each of the three short requests finishes about 11 s sooner.

The patched build passed 39 of my 40 smoke-test questions. The unpatched new build also got 39 on the same 40. OCR was still correct. It's been live on the family server since 2026-11-21.

Test your own server before and after every upgrade

You don't need my patch to use this script. It sends one long prompt, then three short ones 0.15 s later, and prints when each one got its first token and when it finished. It's stdlib only, so there's nothing to install.

Your server needs -np set to 4 or more, or the requests will just queue for a free slot. The chat_template_kwargs line turns off thinking for MiniCPM and Qwen-family models. If your model doesn't accept it, delete that line.

#!/usr/bin/env python3
"""Send one long prompt, then three short ones 0.15 s later, to a llama-server.
Prints how long each short request waits for its first token and to finish.
Usage: python3 mixed_latency.py http://127.0.0.1:8080 [long_words]
"""
import json, sys, time, uuid, urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE = sys.argv[1].rstrip("/") if len(sys.argv) > 1 else "http://127.0.0.1:8080"
LONG_WORDS = int(sys.argv[2]) if len(sys.argv) > 2 else 20000  # roughly 1 token per word

def ask(text, max_tokens):
    body = {
        "messages": [{"role": "user", "content": f"[{uuid.uuid4().hex}] {text}"}],
        "max_tokens": max_tokens, "temperature": 0.7, "stream": True,
        "cache_prompt": False,
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = urllib.request.Request(BASE + "/v1/chat/completions",
                                 data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    t0, ttft = time.time(), None
    with urllib.request.urlopen(req, timeout=600) as r:
        for line in r:
            if not line.startswith(b"data: ") or line.strip() == b"data: [DONE]":
                continue
            choices = json.loads(line[6:]).get("choices") or [{}]
            if ttft is None and choices[0].get("delta", {}).get("content"):
                ttft = time.time() - t0
    return ttft, time.time() - t0

long_text = ("The quick brown fox jumps over the lazy dog. " * (LONG_WORDS // 9)
             + "\nSummarize the text above in one sentence.")
with ThreadPoolExecutor(4) as pool:
    long_job = pool.submit(ask, long_text, 128)
    time.sleep(0.15)
    shorts = [pool.submit(ask, "List three ways to save water.", 64) for _ in range(3)]
    for i, s in enumerate(shorts):
        ttft, total = s.result()
        print(f"short {i}: first token {ttft:.2f} s, done {total:.2f} s")
    print(f"long:    done {long_job.result()[1]:.2f} s")

Here's my output on the live patched server with the default 20,000-word long prompt:

short 0: first token 1.48 s, done 5.75 s
short 1: first token 1.48 s, done 5.75 s
short 2: first token 1.48 s, done 5.75 s
long:    done 13.25 s

Rule of thumb: if a short request waits more than 10 s for its first token, or finishes close to when the long one does, your server is blocking. Mine finished its short requests 7.5 s ahead of the long one.


Deep dive: test setup, the draft model, and rollback

You don't need this section to use the patch. It's here for anyone who wants to reproduce the numbers or hand them to an AI along with their own setup.

Measurement conditions

The box is an EPYC with three modded 2080 Ti 22G cards. These tests used one card. The services on the other two kept running. Before stopping the family server, I rerouted the news pipeline and confirmed 60 s of zero traffic.

The old build is tag b11588, commit 7f51953. The model is MiniCPM-V-4.7-35B-A3B, a 15.5 GB Q3 mixed quant with a merged LoRA, plus the vision mmproj. I used the production flags:

-np 4 -c 131072 -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 -fa on --kv-unified

Requests went through /v1/chat/completions with thinking off, temperature 0.7, top_p 0.95, top_k 20, and max_tokens 512. In the mixed test, short requests used max_tokens 64 and the long one 128. Each request started with a random string and had cache_prompt off, and every request reported cached_tokens 0.

Decode speed is (completion tokens − 1) ÷ (last token time − first token time). That keeps prefill out of the number. Single-stream tests ran 3 times, 4-stream tests ran 2 rounds, and I report medians.

The b12595 + patch run was a separate session later the same day. I compared it with that morning's numbers for the unpatched new build. I didn't rerun the old build.

Why 102.89 tok/s here and 111.5 tok/s in part 25. Part 25 used llama-bench, single stream, with no mmproj loaded. This post uses the production config through the API: 4 slots, q8_0 KV cache, vision loaded. Don't mix numbers across the two posts.

The same goes for latency. Part 25's 3.2–3.8 s was how long a short request waited for its first token under a 90K-token prompt. This post's 7.3 s is the total time to finish 64 tokens under a 30K-token prompt.

Why the long request gets 1.5–1.8 s slower

While other requests are generating, the patch lets the long prompt read only 64 tokens per round. That's the point: the GPU spends the rest of each round on the short requests.

The long request went from 17.99 s on the unpatched new build to 19.53–19.77 s with the patch. The old build took 22.55 s, so it's still faster than before the upgrade.

If only one person uses your server, the patch doesn't help. Leave the variable unset and keep the 17.99 s. For all-long-document batch jobs, the second rule makes concurrent prefills split the budget. I haven't measured that case.

DSpark draft model: still off on the new build

Speculative decoding uses a small draft model to guess the next few tokens, and the big model checks the guesses in one pass. When the guesses are right, you get several tokens for the price of one. I retested MiniCPM-V's official DSpark draft (Q8_0, 424 MB) on b12595. Look at the Chinese 2-stream column: that's my family's normal load.

Draft settingChinese 1 streamChinese 2 streamsEnglish 1 streamCode 1 streamVRAM peak
No draft105.6582.31105.54105.7117.8 GiB
n-max 2100.6163.20121.62120.3220.7 GiB
n-max 389.84not measurednot measurednot measured20.9 GiB
Default89.65not measurednot measurednot measured20.9 GiB

At n-max 2, English single stream gets 15% faster and code 14%. Chinese 2-stream drops 23%, and it costs 2.9 GiB more VRAM. At n-max 3 and at the default, Chinese single stream was 15% slower than no draft. That's also 13% slower than the old build, past my early-abort line, so I stopped those runs.

The new default for --spec-draft-n-max is 3. In October it was 7. My family server is mostly Chinese and multi-user, so it runs with no draft.

My SHA-256 check still points at the old binary

A pre-start script, verify.sh, checks the SHA-256 of llama-server. It hard-codes the old binary's path. So the new binary wasn't blocked, but it also wasn't checked. I need to record a new hash for the new build.

Downtime and rollback

Trying the unpatched new build took the family server down for 17.75 minutes. The patched build trial took 3.96 minutes.

To switch builds, I add a systemd drop-in that points the service at the new binary. To roll back, I delete the drop-in and restart the service.

What I didn't test

  • I only ran the first 40 questions of the 140-question eval set (all GSM8K) on the patched build.
  • The mixed-load test covers one scenario (30K long + 3 short), with 1–3 rounds per build.
  • I only built the patch on e46069c40. Newer master is untested.

Also in this series: #25 MiniCPM-V 4.7 35B-A3B on One RTX 2080 Ti: 111.5 tok/s with Vision · #29 Qwen3.8-27B on One RTX 5090 With QEngine · #21 Dual RTX 2080 Ti FastLLM: No More Head-of-Line Blocking

FAQ

Does llama.cpp have chunked prefill built in?
Not as of November 2026. Upstream chunked prefill is PR #10718, opened in December 2024 and still open. Stock llama-server lets one prompt fill the whole batch (2048 tokens in my config) before it moves on, so a 30K-token prompt takes several rounds and short requests queue behind it. If you want it, you carry a patch yourself.
What happens to my local llama.cpp patches when I upgrade?
They're gone unless you re-apply them, and nothing tells you. An environment variable that only your patch reads is silently ignored: no error, no warning. My LLAMA_PREFILL_CHUNK=64 stayed set on b12595 and did nothing, so short requests queued behind a long prompt went from 7.3 s to 16.8 s. Re-applying the 35-line patch brought them down to 5.4 s.
How do I test whether my llama-server blocks short requests behind a long prompt?
Send one long prompt, then three short ones 0.15 s later, and time each short request's first token and finish. The stdlib-only Python script in this post does exactly that; the server needs -np 4 or more. If a short request waits over 10 s for its first token, or finishes close to when the long one does, it's blocked.

Read next

Don't miss the next one

Subscribe, and you won't.

One-click unsubscribe anytime.