Is Q8 faster than FP16 on an RTX 5090?

For this Qwen2.5-7B single-stream test, Q8_0 generated at 188.75 tok/s versus FP16's 109.67 tok/s. Prompt processing was slightly slower: 15,953.6 versus 16,447.2 tok/s. These are separate measurements at a fixed 512-token depth.

Should I choose Q8_0, Q6_K or Q4_K_M?

I would start with Q8_0 when memory permits, use Q6_K to save more memory, and test Q4_K_M when speed or memory matters most. Q8_0 stayed closest to FP16's output distribution in this English corpus sample. Check your own tasks before treating that as an accuracy guarantee.

[Benchmark] FP16 vs Q8 on RTX 5090: Qwen2.5-7B Generates at 188.75 tok/s

TL;DR

On one RTX 5090, Qwen2.5-7B in Q8_0 generated at 188.75 tok/s, versus FP16's 109.67: 1.72× the speed. Its sampled whole-GPU peak was 10,467 MiB, including the desktop. I would start with Q8_0 for a close FP16 approximation; Q6_K saves more memory, and Q4_K_M prioritizes speed and size. Prompt processing and task accuracy need separate checks.

FP16, Q8_0, Q6_K and Q4_K_M compared on one RTX 5090

Choosing between FP16 and Q8

I wanted to know which file to download: if FP16 already fits, what do I get by choosing Q8 instead? I took one FP16 GGUF of Qwen2.5-7B-Instruct, made three quantized copies, and measured all four on the same RTX 5090.

The quantization introduction explains the names. Here the decision comes from measured speed, memory use and a next-token prediction test. The results apply to this model, build and short context; they do not establish a universal ranking for every GPU.

Q8_0 generates faster; FP16 processes this prompt faster

Read the two speed columns separately. Prompt processing is the model reading the input; generation is the model producing its answer. A token is one of the text pieces the model works with. Both columns count tokens per second, and higher is faster.

The table gives medians of three measured runs. VRAM is the median of each run's sampled whole-card peak, including Windows and desktop applications.

Format	Prompt tok/s	Generation tok/s	Sampled whole-GPU peak (MiB)
FP16	16,447.2	109.67	16,877
Q8_0	15,953.6	188.75	10,467
Q6_K	13,175.0	231.49	8,835
Q4_K_M	15,407.2	284.71	7,471

For generation, Q8_0 reaches 1.72 times FP16's speed. Q4_K_M reaches 284.71 tok/s. For prompt processing, FP16 leads, followed by Q8_0, Q4_K_M and Q6_K. Downloading the smaller file did not make every stage faster.

Separate prompt and generation speed bars for the four formats — Measured medians; each panel has its own scale. All four formats use the same fixed settings.

The quantized weights require fewer bytes to store. Generation speed can benefit from moving less weight data, but this experiment does not isolate the GPU kernels well enough to attribute every speed difference to memory traffic. In particular, Q6_K's prompt result is a measurement, not proof of a specific bottleneck.

Q8_0 stays closest to FP16's output distribution in this sample

All four formats were tested on the first 32 chunks of one English Wikitext-2 test file, with a 512-token context. Perplexity measures next-token prediction: lower is better on this fixed text. KL divergence compares the full predicted token distributions with a saved FP16 reference: closer to zero means closer distributions. Neither metric measures how many coding or math questions the model answers correctly.

Read the perplexity difference against the direct FP16 run, and KL against the stored FP16 logits.

Format	Perplexity	Δ vs direct FP16	KL vs stored FP16 logits
FP16	6.098206	+0.000000	-0.000010
Q8_0	6.106598	+0.008392	0.001709
Q6_K	6.094548	-0.003658	0.004940
Q4_K_M	6.308980	+0.210774	0.039512

Q8_0 has the smallest measured KL among the quantized formats. Q6_K's perplexity is slightly lower than FP16's on this sample, while its KL is higher than Q8_0's. That is not evidence that Q6_K is a better model: predicting the observed next word and matching the entire FP16 distribution are different tests, and this sample is short.

The FP16 self-comparison reports KL −0.000010. True KL cannot be negative; this is a numerical estimate near zero. The stored reference is lossy: the tool reports reconstructed base perplexity 6.085444, whereas direct FP16 perplexity is 6.098206. I keep those values visible rather than force the FP16 KL to zero or mix the two perplexity baselines. The upstream perplexity notes describe the stored-logit representation.

Start with Q8_0; move down when the memory savings matter

My choice for this model is Q8_0 when it fits, Q6_K to use less memory with modest distribution drift, and Q4_K_M when memory or generation speed takes priority. Keep FP16 when you need the unquantized comparison reference, or when prompt processing is the workload you care about most.

These are format choices for one model. I did not measure long-context use, multiple simultaneous requests, Chinese perplexity, or task accuracy. A low KL on English text cannot settle those questions.

Deep dive: reproduce the measurements and inspect the receipts

You can skip this section and still use the recommendations above.

One FP16 file produced all three quantized files

The source is bartowski's FP16 GGUF at a pinned revision. I verified its SHA256 against the repository's LFS object before quantizing. Q8_0, Q6_K and Q4_K_M were generated locally from that file with the same executable and no importance matrix. This avoids mixing prebuilt files with different quantization recipes.

The exact commands below run from my experiment directory, with binaries extracted under bin. The FP16 input is named model-f16.gguf. For each subsequent speed run I changed only -m to the corresponding quantized file.

.\bin\llama-quantize.exe model-f16.gguf model-Q8_0.gguf Q8_0 8
.\bin\llama-quantize.exe model-f16.gguf model-Q6_K.gguf Q6_K 8
.\bin\llama-quantize.exe model-f16.gguf model-Q4_K_M.gguf Q4_K_M 8

.\bin\llama-bench.exe -m model-f16.gguf -p 512 -n 128 -d 512 -b 512 -ub 512 -t 8 -ngl 99 -fa on -ctk f16 -ctv f16 -r 1 -o json

Condition	Value
Date	2026-11-11, Asia/Taipei
GPU / driver	RTX 5090, Windows WDDM, NVIDIA 621.14
CPU / OS	Ryzen 9 9950X; Windows 11 build 26300
llama.cpp	b12250, commit `ae4893a092e060cfbd4979600bf3c1e48a710874`, Windows CUDA 13.5 release
Model parameters	7,615,616,512, from llama-bench metadata
Workload	Single stream; synthetic tokens; prompt 512, generation 128, depth 512
Batch / microbatch / threads	512 / 512 / 8
GPU / attention / KV	`-ngl 99`, Flash Attention on, K and V both F16
Repetition	Four outer rounds; discard round 0; median and range of rounds 1–3; `-r 1` per invocation
Other bench defaults	Built-in warmup on; load/lazy mode auto; repack on; no fit target; full JSON retains the remaining defaults

The format order alternated forward and reverse across rounds. The largest measured speed range was 1.32% of its median, below the predeclared 15% threshold for adding runs. This is a descriptive comparison, not a significance test.

Each invocation creates fresh inference state. There is no serving prefix cache. Since the executable version stayed the same, I did not clear compilation caches belonging to other services. ComfyUI's queue was empty before each invocation; the harness waited if it saw another compute process or an elevated idle baseline. Desktop applications remained open, so this is a Windows desktop measurement rather than a headless-card result. llama-bench excludes tokenization and sampling time, so these rates do not measure complete application response speed.

Whole-card VRAM is measured separately from file size

I sampled nvidia-smi memory usage about every 0.2 seconds while each process ran. The recorded idle baselines were 2,379–2,403 MiB. WDDM did not provide useful per-process memory figures, so I report full-card sampled peaks without claiming they are isolated model allocations. Sampling can miss a shorter peak.

Format	Prompt min–max tok/s	Generation min–max tok/s	Whole-GPU peak min–max MiB	File bytes
FP16	16354.69–16553.35	109.604–109.699	16862–16887	15237853600
Q8_0	15950.65–15963.68	188.614–188.756	10442–10467	8098525600
Q6_K	13028.98–13203.40	231.384–231.716	8810–8835	6254199200
Q4_K_M	15323.84–15462.58	284.697–284.744	7410–7471	4683073952

The raw run records contain every warmup and measured result. The summary JSON contains the arrays used for the medians and ranges. Per-run files such as bench-r1-Q8_0.stdout and bench-r1-Q8_0.vram.json are available in the same directory.

Every quality comparison uses the same saved FP16 reference

.\bin\llama-perplexity.exe -m model-f16.gguf -f wikitext-2-raw/wiki.test.raw -c 512 --chunks 32 -b 512 -ub 512 -t 8 -ngl 99 -fa on -ctk f16 -ctv f16 --kl-divergence-base fp16-reference.kld

.\bin\llama-perplexity.exe -m model-Q8_0.gguf -f wikitext-2-raw/wiki.test.raw -c 512 --chunks 32 -b 512 -ub 512 -t 8 -ngl 99 -fa on -ctk f16 -ctv f16 --kl-divergence-base fp16-reference.kld --kl-divergence

I repeated the second command for FP16, Q6_K and Q4_K_M by changing only the model filename. All runs use the first 32 chunks; this is not the full Wikitext-2 test set. The raw corpus SHA256 is 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08. The saved reference is 2,481,815,316 bytes, SHA256 3c765b5a27dc4dca897d628d3fb3b2c1b386b36243071424345872cf0e1d88f0.

The quality logs retain a tokenizer warning that a control-looking token was assigned a non-control type and was overridden by the loader. Every format inherits tokenizer metadata from the same FP16 source and uses the same loader. I did not manually change the tokenizer.

FP16 quality output, Q8_0 output, Q6_K output and Q4_K_M output retain the chunk results, uncertainties and KL statistics. These uncertainties describe this corpus calculation; they are not repeated-run speed ranges.

Model file SHA256 values identify the exact inputs

FP16: 863c978275bca3fdacdde06cbfc1a65f2bd65210bb7ecfd9ded3b24219d81b54
Q8_0: 6925c96c0e9a6af97a81521caf2dd58da9f0fc5f00620c4d131684b872a6908e
Q6_K: 9616f5a674b8d53dd2c8db918e4d8b5f331a4cca342ba2040c47371a73faecf9
Q4_K_M: 246ac71c280bd514575413b01abef9954f9148c011e4a06df643001a0a6b1910

The manifest also records corpus, reference and executable hashes. The Windows .exe launchers load implementation DLLs; a launcher hash alone is not a complete build identity. The pinned release and CUDA backend hash belong with it.