[Junk-Tier Big Models #4] A frontier-class open model on hardware you already own: DeepSeek-V4-Flash-0731 on one 22GB 2080 Ti

Introduction

You don't stack an entire warehouse against the loading door. Things you handle every day sit by the door, things you handle once a year go to the back, and you accept the walk to the back when it happens. Running a 91GB model on a 22GB graphics card is that arrangement, with VRAM as the space by the door.

Parts 1 through 3 of this series were reports: I ran a big MoE on junk-tier hardware and wrote down what happened. This one is the install guide. Same box, new official release, and this time the goal is that you can reproduce it on whatever card you own.

Everything below was measured on one modded RTX 2080 Ti in an EPYC box. The main deliverable is not my numbers — it's a formula that turns your VRAM into the one flag you actually have to decide.

The 0731 release changed the weights, not the architecture

DeepSeek-V4-Flash-0731 is the official release, published 2026-07-31. It supersedes the preview version and ships substantially stronger agentic behaviour.

I compared the two config.json files field by field. If the architecture had changed, I would have needed to rerun every measurement in this article.

	Preview	0731
Layers (num_hidden_layers)	43	43
Routed experts per layer	256	256
Experts per token	6	6
Shared experts	1	1
max_position_embeddings	1,048,576	1,048,576

Identical. What changed is the weights, not the design — so if you had a working -ncmoe setting for the preview, it carries over.

The reason to spend an evening installing it is in the official card's own comparison table, where DeepSeek puts 0731 next to Opus-4.8:

Benchmark	DeepSeek-V4-Flash-0731	Opus-4.8
Agents' Last Exam	25.2	25.7
Terminal Bench 2.1	82.7	85.0
AutomationBench Public	25.1	27.2
DSBench-Hard	59.6	71.7

The first three are close. The last one is not, and I put it in the table because picking only the flattering rows is how these comparisons usually get written. DeepSeek's own wording is "broadly competitive with the strongest proprietary models available" — competitive, not ahead.

That's enough for the purpose of this article: what you are about to install is not a toy. It runs on your own machine, with no monthly fee and no API bill, and it still works with the network off.

There is one practical difference for anyone self-hosting, and it isn't in that table. This release ships no Jinja chat template. Instead the repo provides an encoding folder of Python scripts and test cases. The consequence is worth sitting with: the chat template inside whatever GGUF you download was written by whoever quantized it. I dumped the template out of my running server and its first line is literally this:

{#- Unsloth template fixes #}

So switching between different people's GGUFs changes more than quantization precision. It changes a template that has no official version to diff against. That's why I stay on the Unsloth build — not because it must be the best one, but because I want a build that runs stable and whose provenance I know.

One number to flag before it confuses you later. HuggingFace shows 304B for this model — that figure comes from the auto-generated safetensors metadata, not from anything written in the model card — while my server reports 284334567511. Part of that gap is that llama.cpp's conversion does not carry the attached speculative-decoding module (num_nextn_predict_layers: 1 in config.json). I did not reconcile the remainder, and I'm not going to guess at it here.

Why a 22GB card can hold a 91GB model

A dense model reads every weight for every token. 91GB of weights would mean moving 91GB per token, and a 22GB card cannot hold that under any arrangement of flags.

This is a Mixture-of-Experts model. Each layer keeps 256 experts, but each token wakes only 6 of them, plus one shared expert that runs every time. That splits the weights into two piles with completely different access patterns:

• Hot — attention, the shared expert, embeddings, the output layer. Small, and read on every single token.
• Cold — the 256 routed experts. This is the bulk of the 91GB, and any given token touches 6 of them.

The hot pile goes in VRAM. The cold pile goes in system RAM, where the CPU computes it. That is the entire trick.

Put another way: a GPU is expensive because it reads fast, and read speed is only worth paying for on data you read every time.

llama.cpp's flag for drawing that line is -ncmoe N (long form --n-cpu-moe N): keep the expert weights of the first N layers on the CPU. The denominator is the model's layer count, which is 43 here.

Setting	What it does	Cost
-ncmoe 43	All experts on CPU	Cheapest on VRAM, slowest
-ncmoe 38	38 layers of experts on CPU, 5 on GPU	What I run on 22GB
-ncmoe 0	All experts on GPU	Needs roughly 90GB of VRAM

Smaller N means more layers on the GPU, which means faster, which means more VRAM. So the whole configuration problem collapses into one question: how many layers can your card take?

Fitting the weights is not the same as running

Before answering that, one very reasonable misconception has to go.

Most people start with a simple subtraction: model size minus VRAM. If the numbers come out close, they assume it will fit. It won't. Your VRAM has to hold three things at once:

1. Model weights — the expert layers you moved onto the card, plus the hot pile.
2. KV cache — everything the conversation has said so far. Whatever you pass to -c sets how big this gets, and that number is yours, not the model's.
3. Compute buffers — scratch space during evaluation, and it grows when you prefill a long input.

Which makes "the weights just fit" a meaningless sentence on its own. If the weights fill the card, the context has nowhere to live. The space you leave for the KV cache is the context length you get.

It also explains why the formula in the next section starts at 11 GB rather than at zero: that floor is not weights, it already contains a full 1M-token KV cache.

Pick your -ncmoe with (VRAM − 11) ÷ 2

I swept -ncmoe on a single card and recorded VRAM at every setting. The result is linear enough to write down as a formula.

Two constants come out of the sweep:

• Baseline 11 GB — VRAM used with all experts on CPU (-ncmoe 43). That already includes a full 1M-token KV cache. It's the unavoidable entry cost.
• About 2 GB per layer — what one layer of experts costs when you move it onto the GPU.

Layers you can move = (your VRAM in GB − 11) ÷ 2 -ncmoe = 43 − (layers you can move)

Which gives, for cards people actually own:

Card	VRAM	Layers on GPU	-ncmoe
RTX 3060	12 GB	0	43 (all on CPU, only just fits)
RTX 4060 Ti	16 GB	2	41
Modded 2080 Ti	22 GB	5	38
RTX 3090 / 4090	24 GB	6	37
RTX 5090	32 GB	10	33

For my 22.5GB card the formula gives 5 layers and -ncmoe 38, which is exactly the boundary I measured. The formula was derived from the sweep, not reverse-engineered to land on an answer I already had.

Here is the sweep it came from — 1M context, median of 3 runs per cell:

-ncmoe	Experts on CPU	VRAM	Decode	Result
43	43/43	11,038 MiB	15.40 tok/s	runs
40	40/43	17,248 MiB	16.07 tok/s	runs
38	38/43	21,184 MiB	16.50 tok/s	what I run
36	36/43	—	—	won't load, short by 1,512 MiB
34	34/43	—	—	won't load, weights alone want 24,394 MiB

The 2GB-per-layer rule holds in the failures too, which is how I trust it. 34 layers on the GPU wants 24,394 MiB, 30 wants 32,266, 24 wants 44,762 — between 1,970 and 2,080 MiB per layer throughout. Even when llama.cpp refuses to load, it tells you exactly how much it wanted. That's a free way to calibrate this formula against a different model or a different quantization: try an aggressive setting, read the number in the error, divide.

Two out-of-memory errors, two opposite fixes

Both failures above print "out of memory". They are not the same problem and they do not have the same fix.

-ncmoe 36 dies like this:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1512.00 MiB on device 0: cudaMalloc failed: out of memory
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache

-ncmoe 34 never gets past the weights:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 24394.40 MiB on device 0: cudaMalloc failed: out of memory
llama_model_load: error loading model: unable to allocate CUDA0 buffer

24,394 MiB requested, on a card that has 22,528 MiB. It was never going to fit.

The last words of the message tell you which resource ran out:

• Ends in kv cache → the weights fit and only KV space is short. Shrink the context, or quantize the KV one step further. Do not touch -ncmoe.
• Ends in unable to allocate CUDA0 buffer → the weights themselves don't fit. Raise N. Shrinking the context will do nothing at all.

I treated both as the same error at first and spent a while tuning in the wrong direction. The tell was in front of me the whole time, and it wasn't the message text — it was the requested number. Wanting 1,512 MiB and wanting 24,394 MiB are not the same kind of problem, and no amount of context tuning closes an 1,866 MiB gap in the weights.

So what does 1M context actually cost?

I expected this to be the expensive part. Same -ncmoe 38, nothing changed but the context length:

Context	VRAM	Decode
65,536	18,874 MiB	16.48 tok/s
262,144	19,348 MiB	16.46 tok/s
1,048,576	21,184 MiB	16.50 tok/s

Sixteen times the context costs 2,310 MiB and no measurable speed change. I assumed I had measured it wrong and ran it again.

The explanation is in the GGUF metadata. attention.head_count_kv is 1, not the usual 8, and attention.sliding_window is 128. This is hybrid attention: a local branch that looks back 128 tokens, long-range entries kept in compressed form rather than stored in full, and a KV cache quantized to 4-bit on top. Those mechanisms stack, which is why the cache doesn't grow proportionally to context. It is not simply that most layers only see 128 tokens.

Practical advice: on this model, set the context length to the maximum. The 2 GB you would save by running 64K is worth about one extra -ncmoe layer, which is worth about 1.4%.

A correction to something I got wrong first time round. The metadata says rope.scaling.type = yarn, rope.scaling.factor = 16.0, rope.scaling.original_context_length = 65536, and I originally read that as proof 1M was raw extrapolation with no training behind it. That inference was wrong. DeepSeek's technical report says V4's pretraining length was extended all the way to 1M and that the model natively supports it; the YaRN parameters describe positional scaling relative to 64K, not an untrained window. The caveat that survives is narrower: I verified that it loads and runs stably at 1M, and I did not measure long-context quality.

Decode is capped by memory bandwidth, not by the graphics card

Filling the VRAM takes decode from 15.40 to 16.50 tok/s. That's +7%, it costs nothing, so take it.

But understand why it's only 7%, or you will spend money in the wrong place. Even at -ncmoe 38, thirty-eight layers of experts still live in system RAM and get read on every token. This box is 8-channel DDR4-2133, roughly 136 GB/s. That is what caps decode. Not the GPU.

Three ways to spend, with very different returns:

Option	Return
Fill the VRAM you already have	Free 7%. Always do it.
Buy a second identical GPU	Barely helps. Turns 5 GPU layers into 10 — another 7% or so — and part 3 measured that multi-GPU also pays a small communication tax.
Move to a box with more memory bandwidth	The actual fix. Same model on DDR5 or a wider platform goes as fast as that path is wide.

On this box, decode speed is set by memory bandwidth, not by the graphics card. Knowing that is what tells you when to stop tuning.

Downloading: the repo has thirteen quants, so scope the include

What you need on the receiving end:

	This box
CPU	AMD EPYC 7402, 24 cores / 48 threads
RAM	123 GiB usable, DDR4-2133, 8 channels
GPU	ONE modded RTX 2080 Ti 22GB (Turing, sm_75), about US$350 second-hand
Disk	NVMe, model is 91GB — leave ~100GB free
OS	Ubuntu, CUDA 12.4, gcc 15.2

At runtime it sits at 70.3 GiB resident, and Cached climbs to 106 GiB because the weights live in page cache via mmap. If your box only has 64GB of RAM, don't try this quant — look at the UD-IQ2 tiers instead.

The Unsloth GGUF repo keeps each quantization in its own folder, so scope the download or you will pull all thirteen of them.

pip install -U "huggingface_hub[cli]" hf_transfer
hf auth login                      # paste the token; never pass it as a command-line argument

export HF_HUB_ENABLE_HF_TRANSFER=1 # without this, 91GB takes a very long time
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
  --include "UD-Q2_K_XL/*" \
  --local-dir ~/models/ds4-0731-ud-q2-k-xl

You end up with three files, and the first one is 5MB:

DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf       5.3 MB
DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00002-of-00003.gguf      49.4 GB
DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00003-of-00003.gguf      47.4 GB

That first file has n_tensors = 0 and holds only metadata. It is not a broken download. I re-downloaded it once before I worked that out. Point -m at the 5MB file and llama.cpp picks up the other two shards itself.

There's a useful side effect: the whole spec sheet lives in that 5MB file. You can read block_count (43) and sliding_window out of it without downloading 91GB first. That's where the layer count in the formula comes from, and it's how you'd calibrate the same formula for a model you haven't pulled yet.

Build llama.cpp yourself, and tell it your GPU generation

deepseek4 is a recent architecture and distro packages of llama.cpp usually don't know it yet. Build from source, and specify your GPU generation while you're at it.

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp-0801
cd ~/llama.cpp-0801
git checkout b10217          # the tag I used, commit ddd4ec1

cmake -B build -DGGML_CUDA=ON \
      -DCMAKE_CUDA_ARCHITECTURES=75 \
      -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

75 is Turing, the 2080 Ti generation. Ampere (30-series) is 86, Ada (40-series) is 89, Blackwell (50-series) is 120.

Then confirm which CUDA it actually linked against, which is not always the one you think you installed:

./build/bin/llama-server --version
ldd build/bin/libggml-cuda.so | grep -E 'cudart|cublas'
# libcudart.so.12 => /usr/lib/x86_64-linux-gnu/libcudart.so.12

This box is CUDA 12 throughout and the build links CUDA 12. Consistent. On a machine with several CUDA installs those two can disagree: the compiler uses the new one, the binary links the old runtime, and the symptom is that the model loads perfectly and then explodes on the first inference with an error that looks nothing like a version problem. That one bit me on another machine, and it's the next article.

The systemd unit, flag by flag

# ~/.config/systemd/user/llama-ds4-8093.service
[Unit]
Description=DeepSeek-V4-Flash-0731 UD-Q2_K_XL single-GPU server on 0.0.0.0:8093
After=network-online.target

[Service]
Type=simple
Environment=CUDA_VISIBLE_DEVICES=GPU-4555abe2-11c5-7920-32b9-864ad7f7a989
ExecStart=/home/coolthor/llama.cpp-0801/build/bin/llama-server \
  -m /home/coolthor/models/ds4-0731-ud-q2-k-xl/UD-Q2_K_XL/DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf \
  --alias ds4-deepseek-v4-flash --host 0.0.0.0 --port 8093 \
  -ngl 99 -ncmoe 38 -c 1048576 -ctk q4_0 -ctv q4_0 \
  --parallel 1 --jinja -fa on --metrics --fit off \
  --log-file /home/coolthor/llama-logs/ds4-8093.log
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now llama-ds4-8093.service
loginctl enable-linger $USER      # so it runs when you're not logged in

Four of those flags are worth explaining, because three of them cost me time.

-ngl 99 and -ncmoe 38 are not contradictory. -ngl puts every layer on the GPU; -ncmoe then pulls back just the expert tensors of 38 of them. The hot attention weights stay on the card. You need both — -ncmoe on its own will not put the rest on the GPU.

CUDA_VISIBLE_DEVICES is set to the card's UUID, not its index. Indexes move across reboots and slot changes; UUIDs don't. This box runs a different service on each of three cards, so a wrong index means quietly stealing someone else's card. Find yours with nvidia-smi --query-gpu=index,uuid,name --format=csv.

--fit off disables auto-fitting. With it on, llama.cpp picks values for the arguments you did not specify so that everything fits; an explicit -c is left alone. I turn it off for cleaner measurements: it removes one value I didn't choose.

--parallel 1 keeps a single slot. The default is four with a unified KV cache, so the main context allocation doesn't simply quadruple — but per-slot checkpoints and state still add pressure, and this model is memory-hungry enough that one slot is the simplest thing to reason about.

Verifying that it loaded what you asked for

# 1. alive? (~11s to ready with a warm page cache)
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8093/health

# 2. is context really 1M, or did something quietly downgrade it?
curl -s http://localhost:8093/props | python3 -m json.tool | grep n_ctx

# 3. which model actually loaded, and how many parameters?
curl -s http://localhost:8093/v1/models | python3 -m json.tool | grep -E 'n_ctx|n_params'

Correct output: the second returns 1048576, the third returns 284334567511. If the second one comes back smaller than you asked for, that's --fit making a decision on your behalf.

Measuring speed needs no extra tooling. The response carries its own timings:

curl -s http://localhost:8093/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"ds4-deepseek-v4-flash",
       "messages":[{"role":"user","content":"Explain in one paragraph what a mixture-of-experts layer does during inference."}],
       "max_tokens":256,"temperature":0.7,"seed":42,
       "chat_template_kwargs":{"thinking":false}}' \
  | python3 -c "import sys,json;t=json.load(sys.stdin)['timings'];print('decode %.2f tok/s'%t['predicted_per_second'])"

What I get: decode 16.4–16.5 tok/s, with no difference between English and Chinese. From real traffic logs, prefill runs 74–78 tok/s on short inputs and 51.6 tok/s on a roughly 5,000-token input, where decode was still 15.49. Decode barely moves as context grows, which is a comfortable property to have in a model you plan to feed long documents.

Gotchas that cost me time (skip this if it already works)

Nothing in this section changes the commands above. Everything works without reading it.

Ignore llama.cpp's own advice about mmap. It prints tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance. Don't take it here. Without mmap, 91GB becomes non-reclaimable resident memory, a 123 GiB box gets very tight, and one large prefill can get the process OOM-killed. With mmap the weights sit in page cache: slower, but it doesn't die overnight.

Never kill by process name on a multi-seat box. All three services on this machine are called llama-server. I followed my own old notes, ran pkill -x llama-server, and killed a neighbour's model. I noticed eight minutes later. Record the PID at launch instead:

setsid nohup llama-server ... & echo $! > /tmp/my-seat.pid
kill -TERM "$(cat /tmp/my-seat.pid)"

Or resolve by port. -f solves the problem of matching your own shell; -x does not solve the problem of matching your neighbour. Both match on a name, and names collide.

Cold and warm starts are nothing alike. About 11 seconds when the page cache is warm, far longer the first time while 91GB comes off disk. 503 Loading model means wait, not restart. Restarting just makes it read everything again.

/props doesn't tell you everything. It reports n_ctx but not the -ncmoe you set. Only the launch argv or the log confirms that. Which is the other reason I insist on --fit off: one fewer parameter that can change by itself is one fewer thing to guess at when you're verifying.

Three decisions, and only one of them is hard

Running this comes down to three choices: which quantization to download, what value to use for -ncmoe, and what context length to configure. The third barely needs a decision on this model — open it all the way. The first follows from how much RAM you have. Only the middle one needs thought, and now there's a formula for it.

Underneath that formula is one idea. A GPU buys read speed, and read speed is only worth buying for the weights you read on every token. Keep those on the card and push the once-in-a-while weights out to cheap RAM. That single sentence explains both why 22GB can hold 284B, and why buying a second card won't do much for you.

Next: the same model and the same GGUF on a DGX Spark, where -ncmoe isn't needed at all — but a CUDA version trap is waiting, and it does not look like a version problem.

---

Also in this series:

• #1 Running a 119B MoE at 74 tok/s on Three 2080 Tis
• #2 Running Poolside Laguna S 2.1, a 118B Coding MoE, on ONE 22GB 2080 Ti
• #3 A 284B MoE on ONE 2080 Ti — and it beats my DGX Spark