LFM2.5 24B-A2B on One RTX 2080 Ti: 184.8 tok/s, If You Use the Official Sampling

Why I tested it

In #25 I was looking for a model that could take over my family's two-GPU 27B chat server on a single card. The same night, Liquid AI released LFM2.5 24B-A2B, another MoE with about 2B active parameters, so it went through the same test set.

It isn't a plain Transformer. Only 10 of its 40 layers are attention; the other 30 are short convolutions. Fewer attention layers mean a small KV cache, and few active parameters mean little to read per token. Both make it fast.

It was the fastest model of the night: 184.8 tok/s on one card, tied with the 27B on the 140 set. But thinking can't be turned off, temperature 0 can send it into very long traces, it passed 1 of 5 agent tasks, and it can't read images, so the family chat server went with MiniCPM. The rest covers setup, the numbers, and why the official sampling settings matter. The --kv-unified setting that cost me 12 bogus errors is in the deep dive.

184.8 tok/s on one card, 131K context in 17.4 GB

Per Liquid AI: 24B total, 2.45B active per token, 64 experts per MoE block with top-4 routing, and a 131,072-token context.

Three ways to run it on the same machine:

	1 card, QAD Q4_0 13.6G	1 card, Q4_K_M 14.5G	2 cards, Q8_0
Prefill 512 / 8K / 32K (tok/s)	3243 / 3286 / 2881	2864 / 2959 / 2631	3086 / 4930 / 4659
Decode (tok/s)	184.8	180.0	135.0
Decode at 32K depth (tok/s)	not measured	148.4	113.6
Full 131,072 context	not measured	17.4 GB	14.2 GB per card

QAD is Liquid's quantization-aware-distilled Q4_0. It's 0.9 GB smaller than Q4_K_M and decodes a bit faster. As with MiniCPM, one card decodes faster than two because the weights are smaller; two cards win on long-prompt prefill, roughly 1.5–1.7x at 8K and above.

On a 120K-token needle test, one card took 52 seconds (2,319 tok/s) and two cards 28 seconds (4,256); both answered correctly. MiniCPM needed 154 seconds for 165K tokens on one card, so LFM reads long documents much faster.

These numbers come from one night's setup: llama.cpp b11588, with another GPU running a separate test.

128/140, tied with the two-GPU 27B

Same 140 questions: GSM8K 50, MATH-500 30, MuSR 20, HumanEval+ 40.

	2 cards, Q8_0, official sampling	1 card, QAD Q4_0, Qwen sampling
GSM8K	49/50	47/50
MATH-500	29/30	28/30
MuSR	15/20	14/20
HumanEval+	35/40	36/40
Total	128/140	125/140
Wall time	3,169 s	1,344 s

The two-GPU 27B scored 128. The two columns differ in both quant and sampling, so I can't say which one costs the three points.

I normally run this set with thinking off. LFM can't turn it off, so it thinks on every question; the two-card run took 3,169 seconds, while MiniCPM with thinking off took 576.

Hard suite: better at instructions, worse at code, and some math answers cut off at 32K

Thinking on. One card ran QAD at temperature 0; two cards ran Q8_0 with the official sampling.

	1 card QAD, temp 0	2 cards Q8_0, official	Qwen3.8-27B
Math (AIME)	23/30 (7 truncated)	23/30 (6 truncated)	21/30
Reasoning (GPQA)	21/30	23/30	25/30
Instructions (IFEval)	27/30	29/30	27/30
Code (LiveCodeBench)	9/20 (7 truncated)	11/20 (6 truncated)	14/20

"Truncated" means the thinking trace hit the 32K-token cap before an answer came out, which counts as wrong.

LFM did worst on these two:

• Agent bug fixing (Pi fixing real bugs in a repo, hidden tests in Docker, one card with QAD): 1/5. Its tool calls were well-formed; it just couldn't fix the other four within 20 turns. MiniCPM went 5/5.
• Three-hop lookup over 116K tokens: 0/10. I raised its answer budget to 8,192 tokens (2,048 for other models) because at 2,048 it spent the whole budget thinking on the first question and returned an empty string.

Use the official sampling: temperature 0 can think forever

I run evals at temperature 0 so results are reproducible. With LFM, some questions at temperature 0 never wrap up.

The median reasoning trace was 1,771 tokens, which is fine, but one question ran to 28K. In the one-card run, 7 math and 7 code questions hit the 32K cap.

The model card recommends temperature 0.1, top_k 50, and repetition_penalty 1.05. With those on the two-card run, reasoning went from 21 to 23, instructions from 27 to 29, code from 9 to 11, and truncations from 7 to 6. The quant also changed (Q4_0 to Q8_0), so not all of that is sampling. Truncations still happened, though: even with the recommended settings, hard problems can produce very long traces.

Send the three parameters with each request:

curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "What is 17 × 23?"}],
  "temperature": 0.1, "top_k": 50, "repeat_penalty": 1.05,
  "max_tokens": 8192
}'

Give max_tokens real headroom. The model thinks before every answer, even 17 × 23, and a small budget cuts it off before the answer appears. Adding "chat_template_kwargs": {"enable_thinking": false} does nothing; the model card says it always thinks.

Setup: QAD Q4_0 on one card with 131K context

You need a modded 2080 Ti 22G and a recent llama.cpp. I used b11588, built for sm_75 the same way as in #25.

mkdir -p ~/models/lfm25 && cd ~/models/lfm25
curl -L -C - --retry 5 -o LFM2.5-24B-A2B-QAD-Q4_0.gguf \
  https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF/resolve/main/LFM2.5-24B-A2B-QAD-Q4_0.gguf

Launch:

./build/bin/llama-server \
  -m ~/models/lfm25/LFM2.5-24B-A2B-QAD-Q4_0.gguf \
  -ngl 99 -fa on -t 24 --jinja \
  -c 131072 -np 1 \
  --temp 0.1 --top-k 50 --repeat-penalty 1.05 \
  --host 0.0.0.0 --port 8080

• --temp, --top-k, and --repeat-penalty set server-side defaults for requests that don't send their own.
• To serve several users, raise -c along with -np. The deep dive explains why.
• There's no speculative decoding to turn on: no MTP and no official draft model. N-gram speculation gave 2–5%.

Why it didn't replace the family chat server

Both models finished the same night. LFM decodes over 1.6x faster (184.8 vs 111.5 tok/s) and matched the 27B on the 140 set. I still picked MiniCPM:

• No vision. LFM is text-only; MiniCPM reads images.
• Thinking can't be turned off. Even short questions get a thinking pass first, which eats into the speed advantage.
• 1/5 on agent tasks. That server also handles agent tasks for me, and MiniCPM passed all five.

After testing I deleted the three LFM weight files (53.6 GB) and kept the results.

---

Deep dive: --kv-unified and 12 bogus errors, long-context runs, methodology

You can skip this section. It covers the KV pool trap when serving several users, why long-context questions need their own run, the sampling mismatch in my harness, and the methodology and environment.

With --kv-unified, -c is one shared pool

For the two-card runs I used 8 slots (-np 8), -c 131072, and --kv-unified, assuming each request could use up to 131,072 tokens. Twelve questions came back as HTTP 500 and my runner logged them as errors.

Under --kv-unified, -c is the total KV cache shared by all 8 slots. Each long-thinking LFM request took 30K+ tokens, so four in flight nearly filled the 131,072-token pool and the next one didn't fit.

With -c 393216, the server log reads:

llama_context: n_ctx_seq (393216) > n_ctx_train (131072) -- possible training context overflow
srv    load_model: initializing, n_slots = 8, n_ctx_slot = 131072, kv_unified = 'true'

Each slot is still capped at 131,072 (the training length), and the pool is 393,216. With only 10 attention layers, LFM's KV cache is about 20 KB per token, so a bigger pool is cheap and still fit on the two cards. Rerunning those 12 questions produced the scores reported earlier.

Long-context questions need their own run

A 116K-token question shares that same pool. When I ran the long-context set alongside the hard suite with a 131K pool, one long request took most of the cache and other requests started returning HTTP 500. I moved the long-context set to its own run.

My harness used Qwen's sampling

The 140-question runner defaults to Qwen's recommended sampling. I didn't change it for the one-card run, so 125 was measured with Qwen sampling; for the two-card run I switched to Liquid's settings and got 128.

Methodology

Speed: llama-bench with -ngl 99 -fa 1 -t 24, prompts of 512, 8192, and 32768 tokens, 128 generated tokens; the 32K-depth column adds -d 32768.

Context and needle: llama-server -c 131072, VRAM read from nvidia-smi; the needle prompt is about 120K tokens of random English words with a passphrase planted inside.

140 questions: two cards Q8_0 with the official sampling, one card QAD Q4_0 with Qwen sampling; HumanEval+ scored with evalplus.

Hard suite: thinking on, 32K cap per question. One card QAD at temperature 0; two cards Q8_0 with the official sampling (temperature 0.1, top_k 50, repeat_penalty 1.05).

Agent: one card, QAD; Pi fixes bugs in a real repo, up to 20 turns, hidden tests in Docker decide pass or fail.

Three-hop lookup: 116K tokens, 8,192-token answer budget, run on its own.

Caveats:

• Every test ran once, so there are no confidence intervals.
• The 27B scores came from an earlier date under slightly different conditions.
• On the speed-test night, another GPU was running a separate test.

Versions and hardware

• llama.cpp b11588, built for sm_75
• Model: QAD-Q4_0 and Q4_K_M (one card) and Q8_0 (two cards) from LiquidAI/LFM2.5-24B-A2B-GGUF
• Machine: EPYC, 128 GB RAM, three RTX 2080 Ti 22G; one-card tests on the third GPU, two-card tests on the first two (with the 27B stopped)

Not done yet

• The 140 set and hard suite on one-card QAD with the official sampling.
• Rerunning the truncated questions with a 64K cap, to tell long-but-finishing traces from real loops.
• A separate Chinese decode-speed measurement.

---

Also in this series: #25 MiniCPM-V 4.7 35B-A3B on one RTX 2080 Ti · #24 177B on 44 GB of VRAM: two 2080 Tis run Strata at 98.7 tok/s · #22 NVFP4 on a 2080 Ti without FP4 hardware