MiniCPM-V 4.7 35B-A3B on One RTX 2080 Ti: 111.5 tok/s with Vision, Replacing a Two-GPU 27B

Why I tested it

My box has three modded 2080 Tis. The family chat server ran Qwen3.8-27B (#22) across the first two. The third card was shared between ComfyUI and Strata (#23), so they took turns. #24 ended with Strata being great for one user and no card I could give it full-time.

Then OpenBMB released MiniCPM-V 4.7 35B-A3B, the first MoE MiniCPM-V: 35B total, about 3B active per token, and it reads images. If it could replace the 27B on one GPU instead of two, I'd get a card back.

It could. One card runs it, it reads images, the full 262K context fits, and it came within one question of the 27B. I switched the family chat server to it and gave the freed card to Strata. The first half of this post is setup and numbers. Speculative decoding, the three-hop failure, and a prefill fix for mixed long and short requests are in the deep dive.

111.5 tok/s on one card, 262K context in 20.8 GB

Each layer splits most of its weights into experts and picks a few per token, about 3B active in total (OpenBMB's own docs say about 3.5B). The language side is a Qwen-family MoE (qwen3_5_moe in the config); vision comes from SigLIP2, with images going through an mmproj projector file first.

That's a good fit for a 2080 Ti. The official mixed quant mix_q3gud_im is 15.5 GB, so it fits on a 22 GB card with room left over, and each token only reads about 3B parameters, so decode is fast.

Same machine, one card on the mixed q3, two cards on Q8_0:

	1 card, mix_q3gud_im 15.5G	2 cards, Q8_0
Prefill 512 / 8K / 32K (tok/s)	1938 / 1896 / 1597	2213 / 3450 / 3050
Decode (tok/s)	111.5	96.5
Decode at 32K depth (tok/s)	96.0	84.6
165K-token needle test	154 s	82 s

One card decodes faster than two because the q3 weights are smaller and each token reads less. Two cards win on prefill, about 1.8x at 8K and up. Both answered the needle test correctly.

Text-only at 262,144 tokens (the model's limit), one card uses 20.8 GB. Running vision at the same time needs a q8_0 KV cache; the exact config is in the family chat server section below.

These numbers come from one night's setup: llama.cpp b11588, with the other GPUs running a separate test that took some CPU.

127/140, one question behind the two-GPU 27B

My 140-question set is GSM8K 50, MATH-500 30, MuSR 20, and HumanEval+ 40, the same mix as #19. Thinking off, four requests in parallel.

	2 cards, Q8_0	1 card, q3 + LoRA
GSM8K	49/50	49/50
MATH-500	30/30	30/30
MuSR	14/20	14/20
HumanEval+	34/40	36/40
Total	127/140	129/140

The two-GPU 27B scored 128 on the same set with thinking off.

The second column is what actually went into production: the one-card q3 with a community uncensoring LoRA (rank 1, 106 MB). I never ran the plain one-card q3 on the 140, so I can't tell whether the quant or the LoRA accounts for the two-point gap. Vision still worked with the LoRA loaded; the same receipt came back fully correct.

Better at math, worse at multi-step reasoning

The 140 set doesn't separate the two models much, so I also ran a hard suite with thinking on and temperature 0, on two cards with Q8_0:

	MiniCPM-V 4.7	Qwen3.8-27B	Gap
Math (AIME)	25/30	21/30	+4
Reasoning (GPQA)	14/30	25/30	−11
Instructions (IFEval)	24/30	27/30	−3
Code (LiveCodeBench)	12/20	14/20	−2

It solved 4 more AIME problems and 11 fewer graduate-level GPQA questions.

Two more results:

• Three-hop lookup over 116K tokens (follow a chain of clues three times through a long document): 0/10 without thinking, 2/10 with it. The 27B only ran 5 of these and got all 5. MiniCPM fails the same way every time; the deep dive breaks it down.
• Agent bug fixing (Pi fixing real bugs in a repo, scored by hidden tests in Docker): 5/5, 15 to 23 seconds each, on one card with the q3 quant.

Vision: a Traditional Chinese receipt, read correctly

Vision ran on one card with the q3 quant, mmproj-model-f16.gguf, and 16K context. I gave it two images:

• A receipt in Traditional Chinese, asking for every item, price, and the total: 6 items and the 260 total, all correct.
• A generated image of a woman in period costume eating in a convenience store, asking it to describe the scene in Traditional Chinese and point out what doesn't make sense: it caught the oddity, and the whole answer had exactly one Simplified character (处).

Two images is a smoke test, not a vision benchmark. It shows the model reads Traditional Chinese and doesn't drift into Simplified.

Setup: build b11588, grab the 15.5 GB file, disable thinking per request

You need:

• One modded 2080 Ti 22G.
• A recent llama.cpp. MiniCPM-V 4.7 support is PR #29416, merged on 10-10 and included from b11551 on. Older builds won't load the model. I used b11588.
• Weights: MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf (15.5 GB), plus mmproj-model-f16.gguf for vision.

Build for sm_75

The 2080 Ti is Turing (sm_75), so I build from source:

git clone --depth 1 --branch b11588 https://github.com/ggml-org/llama.cpp llama.cpp-b11588
cd llama.cpp-b11588
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j 40 --target llama-server llama-mtmd-cli llama-bench

Set -j to your core count.

Download

mkdir -p ~/models/minicpm-v47 && cd ~/models/minicpm-v47
B=https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf/resolve/main
curl -L -C - --retry 5 -o MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf $B/MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf
curl -L -C - --retry 5 -o mmproj-model-f16.gguf $B/mmproj-model-f16.gguf

-C - resumes; if the download drops, rerun the same line.

Launch: text-only at 262K

This is the exact command behind the 20.8 GB and the passing needle test:

CUDA_VISIBLE_DEVICES=GPU-<your card's UUID> \
./build/bin/llama-server \
  -m ~/models/minicpm-v47/MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf \
  -ngl 99 -fa on -c 262144 -np 1 -t 24 --jinja --fit off \
  --host 0.0.0.0 --port 8080

• I have several GPUs, so CUDA_VISIBLE_DEVICES pins the card by UUID (nvidia-smi -L lists them). Skip it if you have one card.
• --fit off turns off automatic fitting and loads exactly the context you asked for. Older docs say --no-fit, which b11588 rejects.
• -t 24 is my CPU thread count. Adjust for your machine.

Disable thinking per request

The model thinks by default. To turn it off, send chat_template_kwargs with each request:

curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Write a Python function that checks whether a number is prime"}],
  "chat_template_kwargs": {"enable_thinking": false},
  "max_tokens": 512
}'

Images go through the same API as base64 in image_url:

import base64, json, urllib.request

b = base64.b64encode(open("receipt.png", "rb").read()).decode()
body = {
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64," + b}},
        {"type": "text", "text": "List every item and price on this receipt exactly as printed, and the total."},
    ]}],
    "temperature": 0.2, "max_tokens": 800,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request("http://localhost:8080/v1/chat/completions",
                             json.dumps(body).encode(), {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req, timeout=600))["choices"][0]["message"]["content"])

Three things that will trip you up

1. llama-mtmd-cli doesn't accept --chat-template-kwargs. You can't do image prompts with thinking off from mtmd-cli. Run the server and use the API.
2. Skip Q4_K_M. It doesn't fit on one card: you have to push 4 layers of experts to the CPU (-ncmoe 4), leaving 0.8 GB of VRAM, and decode drops to 89 tok/s. The q3 mix stays fully on the GPU at 111.5.
3. Old llama.cpp won't load it. You need a build with PR #29416 (b11551 or later).

The family chat server: vision, two concurrent users, q8_0 KV

This is the config our family chat server runs now:

CUDA_VISIBLE_DEVICES=GPU-<your card's UUID> LLAMA_PREFILL_CHUNK=64 \
./build/bin/llama-server \
  -m MiniCPM-V-4.7-35B-A3B-LoRA1.0-mix-Q3_K.gguf \
  --mmproj mmproj-model-f16.gguf \
  -ngl 99 -fa on -t 24 --jinja --fit off \
  --kv-unified -np 2 -c 131072 -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 512 --metrics --slots

• Weights: I merged the LoRA into the weights at high precision, then requantized to the same layout as the official q3 mix. Still 15.5 GB, and no adapter to load at runtime. I haven't rerun the 140 on this merged file; the 129 was measured with the LoRA loaded at runtime.
• -ctk q8_0 -ctv q8_0: q8_0 KV cache. With the default f16, text-only 262K already takes 20.8 GB and adding the 1.07 GB mmproj runs out of memory. With q8_0, vision plus 262,144 context comes to 20.4 GB. The family chat server runs 131,072 to leave some headroom.
• --kv-unified -np 2: two users at once. With --kv-unified, -c is the total shared by both slots, not a per-request cap.
• LLAMA_PREFILL_CHUNK=64: not an upstream option. It's my own port of a patch that keeps a long prompt from blocking everyone else's short questions. Stock llama.cpp doesn't have it, and leaving the variable unset gives the original behavior. Details in the deep dive.

For more users, I tested 8 slots on the third card (-np 8 -c 262144, official q3 without the LoRA, otherwise the same, 20.4 GB):

Concurrent users	Per user (tok/s)	Total (tok/s)
1	107.5	103.6
2	80.9	154.9
4	50.6	189.5
8	30.2	224.1

After the switch I re-measured a single user on the production card: 99–104 tok/s decode, about 2,000 tok/s prefill, 77 at 32K depth, with the card still capped at 200 W.

With the family chat server on one card, the 27B on the first two cards was retired. The second card now runs Strata full-time and the third stays on ComfyUI. Each card has one job, and nothing has to take turns.

---

Deep dive: speculative decoding, MTP, the three-hop failure, mixed-length prefill, methodology

You can skip this section. It covers why speculative decoding gets slower, why MTP isn't usable, where the three-hop lookup breaks, how I stopped long prompts from blocking short ones, and the full methodology, environment, and gotchas.

Why does the draft model make it slower when VRAM is fine?

OpenBMB ships a separate DSpark draft model for speculative decoding (a small model guesses a few tokens ahead and the main model verifies them in one pass). With the official default n-max 7, Chinese output fell from 110 to 43 tok/s.

My first guess was VRAM, with the draft crowding the KV cache. With the draft loaded the card used 18 of 22 GB, so that's not it.

Then I swept settings on one card with the q3 quant, temperature 0, across Chinese, code, and English prompts:

Setting	Chinese	Code	English
Official default, n-max 7	110 → 43 tok/s
n-max 2	−25% (28% accepted)	+20% (76% accepted)	+4%
n-max 2 + --spec-draft-p-min 0.75	99 → 61 tok/s

(The n-max 7 row was a separate run with a no-draft baseline of 110; the p-min 0.75 row had a baseline of 99.)

n-max 3 and 4 were slower than 2. Chinese acceptance sat at 15–28%, so most guesses were thrown away.

The third row is the telling one. With --spec-draft-p-min 0.75 the draft only submits when it's confident: 45 drafts across the whole Chinese answer, 96% accepted, and it still ran at 61 tok/s against 99 without a draft. With acceptance that high, guess quality isn't the issue. The cost is the fixed work the draft does at every step. p-min 0.5 was also slower than leaving it unset.

Three things stack up here:

1. Each step is already cheap. A token touches about 3B parameters, which is why one card hits 111.5 tok/s. Speculative decoding saves weight-reading time, and there wasn't much of it.
2. Verifying several tokens saves less on an MoE. A dense model verifying n tokens reads its weights once, which is where the speedup comes from. Here each of the n tokens routes to its own experts, and they don't fully overlap, so a batch reads more expert weights than a single token does and the savings shrink.
3. Chinese guesses miss. At 15–28% acceptance, most of the verification is wasted.

I saw the same pattern with Bonsai 2 in #19: speculative decoding pays off when decode is bound by memory bandwidth.

These flags give the n-max 2 setup. It's only worth it for workloads that are almost all code or English:

-md ~/models/minicpm-v47/dspark-MiniCPM-V-4_7-35B-A3B-F16.gguf \
  --spec-type draft-dspark --spec-draft-n-max 2 -ngld 99

b11588 accepts --spec-draft-p-min. Some docs, including llama.cpp's own docs/speculative.md, still say --spec-draft-conf-min, and copying that fails at load:

error: invalid argument: --spec-draft-conf-min

MTP is in the config but not in the weights

Qwen-family models often ship MTP (built-in draft layers, no separate model needed), and llama.cpp supports MTP for qwen35moe. This model's config.json says mtp_num_hidden_layers: 1.

The weight index on Hugging Face has zero MTP tensors. The config declares it but the weights don't include it, so the only option for now is the external DSpark draft.

Three-hop lookup: finds the trail, stops one hop short

Each question hides a chain inside a 116K-token document: an alias leads to a hub, the hub leads to a vault, and the vault leads to the answer. Miss any hop and the answer is wrong.

On two cards with Q8_0 it scored 0/10 without thinking and 2/10 with it. The misses all look the same: the first two hops are right, it reaches the vault, and then returns the vault's ID as the answer instead of taking the last hop.

That lines up with the 11-question GPQA gap: problems that need a few more reasoning steps are where it trails the 27B.

Long prompts blocking short ones: porting chunked prefill

Before the switch I tested a case that actually happens at home: someone pastes a long document while someone else asks a one-line question.

Stock llama.cpp prefills the whole long prompt before the short one gets a turn. With a 90K-token request and three short ones, the short requests waited 27.5 to 37.5 seconds before their first token.

The fix is to prefill the long prompt in chunks and let other requests in between chunks. llama.cpp has a PR #10718 that does this, still open and written against an older code path. I ported the approach by hand into two spots in b11588's server-context.cpp, with an environment variable LLAMA_PREFILL_CHUNK setting the chunk size; 0 keeps the original behavior.

Chunk size	Short request wait	Long request itself
none	27.5–37.5 s	63.6 s
512	7–8.8 s
256	5–6.2 s
64	3.2–3.8 s	65.5 s (+3%)

At 64, the short questions go from half a minute to just over 3 seconds, the long request only slows by 3%, and every answer was correct. Chunk 32 was within a second of 64, so I use 64.

Three more checks:

• 8 slots, two 60K-token requests plus two short ones: short waits went from 35.7–37.4 s to 4.4–4.5 s. The two long requests now run side by side and both finish around 100 s; without chunking, the first one finished at 49.5 s.
• A short image request arriving during a 90K-token request: 7.1 s down to 3.0 s, answer correct. I didn't touch the vision path, and it benefits too.
• 4 and 8 parallel requests, each with a unique tag: no answer contained content from another request.

Q4_K_M doesn't fit on one card

I originally wanted Q4_K_M as the one-card quant. It doesn't fit: -ncmoe 4 moves 4 layers of experts to the CPU, VRAM is down to 0.8 GB after loading, and decode is 89 tok/s. The official q3 mix stays fully on the GPU at 111.5 with room for 262K, so that's what I use.

Blender scripts: 2/5 bare, 5/5 with a cheat sheet and error feedback

I also have local models write Blender Python scripts, so I made 5 bpy tasks (a booth, a bookshelf, a previs camera move, materials, a chair).

• Bare prompt: MiniCPM with the LoRA 2/5, the 27B 4/5. Both got the Track To constraint enum wrong; MiniCPM also used outdated API calls on the chair and materials tasks.
• With a bpy cheat sheet and the error message fed back for a rewrite (up to 2 retries): both 5/5. The cheat sheet alone didn't help; the error-feedback retry did.

That retry loop is how I actually work, so under my setup this one is a tie.

Methodology

Speed: llama-bench with -ngl 99 -fa 1 -t 24, prompts of 512, 8192, and 32768 tokens, 128 generated tokens, two runs each; the 32K-depth column adds -d 32768.

Context limit and needle: llama-server with -c 262144 -np 1 --fit off. The prompt is random English words filling about 85% of the context, with a passphrase planted at the 60% mark; temperature 0, thinking off, ask for the passphrase. One card and two cards used the same 165,107-token prompt.

Speculative decoding: one card, q3, -c 16384 -np 1, one Chinese, one code, and one English prompt at temperature 0; VRAM read from nvidia-smi 45 seconds after the server loaded.

140 questions: -np 4 --kv-unified, four in parallel, thinking off; HumanEval+ scored with evalplus. Two-card Q8_0 took 576 s, one-card q3 with the LoRA 772 s.

Hard suite: two cards, Q8_0, thinking on, temperature 0: AIME, GPQA, IFEval, LiveCodeBench, and the three-hop lookup.

Agent: one card, q3; Pi fixes bugs in a real repo, hidden tests in Docker decide pass or fail; temperature 0.

Concurrency: third card, official q3 plus vision, q8_0 KV, -np 8 -c 262144 --kv-unified, LLAMA_PREFILL_CHUNK=64, sending 1, 2, 4, and 8 requests at once.

Caveats:

• Every test ran once at a fixed temperature, so there are no confidence intervals.
• The 27B scores came from an earlier date under slightly different conditions.
• On the speed-test night, the other GPUs were running a separate test that used CPU, so absolute tok/s belongs to that setup.
• My first multi-user run on the production card showed 674 tok/s prefill and 41 tok/s decode. Strata was reloading on the same machine and outside requests were coming in, so those numbers were contaminated. For the rerun I recorded the request counter from /metrics and the /slots state before and after to confirm nothing else was hitting the server; that's where the 99–104 comes from.

Gotchas

• b11588 rejects --no-fit: use --fit off.
• b11588 rejects --spec-draft-conf-min: use --spec-draft-p-min; the error is shown in the speculative decoding part of this deep dive.
• Vision plus 262K with an f16 KV cache runs out of memory: use -ctk q8_0 -ctv q8_0.

Versions and hardware

• llama.cpp b11588, built for sm_75 (-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DLLAMA_CURL=OFF); the family chat server runs a copy of the same build with the chunked-prefill port
• Model: mix_q3gud_im (one card), Q8_0 (two cards), and Q4_K_M (speed only) from openbmb/MiniCPM-V-4.7-35B-A3B-gguf; mmproj-model-f16.gguf for vision; dspark-MiniCPM-V-4_7-35B-A3B-F16.gguf as the draft; LoRA from AtomicChat/MiniCPM-V-4.7-35B-A3B-Uncensored-GGUF
• Machine: EPYC, 128 GB RAM, three RTX 2080 Ti 22G. One-card tests on the third GPU, two-card tests on the first two (with the 27B stopped), production on the first

Not done yet

• The 140 set and hard suite on the plain one-card q3 without the LoRA. The only capability scores are two-card Q8_0 and one-card with the LoRA.
• A 140 rerun on the merged-and-requantized file.
• Vision beyond two images.
• My own refusal-rate test.

---

Also in this series: #24 177B on 44 GB of VRAM: two 2080 Tis run Strata at 98.7 tok/s · #23 One modded 2080 Ti runs 177B with Strata · #22 NVFP4 on a 2080 Ti without FP4 hardware · #19 Ternary Bonsai 2 on one RTX 2080 Ti