[Just for Fun] A GTX 970 as an offline voice assistant: Gemma 4 E2B + Piper TTS (2.8s end-to-end)

Come hear a 10-year-old graphics card talk 👇 — this is the actual synthesized reply from the assistant, generated end-to-end on a 2014 GTX 970:

The short version

The first post asked "how fast." This one asks "how far." It turns out a 2014 GTX 970 — the junk-bin card from the last post — can be a complete offline voice assistant. It looks at a photo and tells you it's a cat. It listens to you talk and writes down what you said. It answers, writes Python, and then a tiny CPU-only text-to-speech engine reads the answer back out loud. No cloud, no API key, no subscription. And in my case, no hardware bill at all — it's a PC I'd already retired to the closet. (If you wanted to buy in, a used 970 is about two coffees.)

---

From "how fast" to "how far"

The first post settled the speed question: Gemma 4 E2B runs at ~47.6 tok/s on a GTX 970, and a tensor-core-less card is dequant-bound. But Gemma 4 E2B isn't a text model with a vision bolt-on — it's natively multimodal. It takes images and audio in. The obvious next question for a 3.5GB junk card: can it actually use any of that? And if it can hear, can I make it talk?

This is the second post in the GTX 970 mini-series. Same hardware, same model, no new money — just seeing how much of an assistant fits before VRAM runs out.

Fitting the multimodal stack into 4GB: 942MB of projector, 535MB to spare

Multimodal input rides on a separate file: gemma-4-E2B-it-mmproj.gguf, the projector that encodes images and audio into tokens the language model understands. It ships in Google's QAT Q4_0 repo and weighs 942MB. The math doesn't look promising:

Model (QAT Q4_0):     3,194 MB
Projector (mmproj):     942 MB
Total:                4,136 MB
GTX 970 usable VRAM:  ~4,096 MB (3.5GB fast + 0.5GB slow)

That's over budget. But llama.cpp doesn't pin every weight in VRAM at once, so it loads anyway — add --mmproj to the server command:

./build/bin/llama-server \
  -m ~/models/gemma-4-E2B_q4_0-it.gguf \
  --mmproj ~/models/gemma-4-E2B-it-mmproj.gguf \
  -ngl 99 -c 2048 --port 8080 \
  --jinja --chat-template-kwargs '{"enable_thinking":false}'

Measured VRAM: 3,405MB idle, 3,496MB mid-inference — 535MB to spare. Tight, but stable across every test, no OOM. One caveat: llama.cpp lists audio input as "highly experimental." It worked end-to-end for me, but it's newer and rougher than the vision path.

It sees, and it hears

Vision. Here's the photo I fed it, with "What is in this image?":

Q4_0 nailed it: a tabby cat, lying down, looking at the camera, blurred indoor background. In Traditional Chinese it did the same in three clean sentences. A salad photo came back with the individual ingredients — lettuce, red onion, cucumber, carrots, peppercorns:

The one gotcha I hit: putting the image_url block after the text made the model insist it got no image. Google recommends image-before-text (and audio-after-text); ordering it that way fixed it.

Audio. The same projector handles speech. Gemma 4 E2B takes up to 30-second clips. Here's the English clip I fed in:

Transcription was near-perfect — it came back missing only an "a" from "Hello, this is a test of speech recognition on a GTX 970 graphics card." And the Traditional Chinese clip:

It got the front half perfect, then misheard 「語音辨識」 (speech recognition) as 「與音變式」. I can't pin that on the model, though: the input was synthetic TTS, so the test can't separate a model error from an artifact in the generated speech.

"Multimodal is faster" — careful, that's mostly measurement

Here's the trap I almost walked into. The image and audio runs clocked higher than text: ~51–53 tok/s versus 47.6 for pure text. It's tempting to write "multimodal is faster because the projector compresses media into fewer tokens."

That explanation doesn't hold up — at least not as stated. Fewer input tokens make prefill, the prompt-processing pass, cheaper, and a shorter context can lighten the KV-cache and attention work a little, so it's not impossible for decode to nudge up. But that's a long way from "multimodal is faster." Far more likely, the higher reading is prefill bleeding into the average or just a shorter run, not the generation phase speeding up. Without separating prompt-eval from generation timing, you can't call it a real decode speedup. The model isn't generating faster because it's looking at a cat instead of reading a sentence — believe the mechanism, not the convenient number.

Giving it a voice: Piper TTS, on the CPU, for free

The model can listen. To make it talk I needed text-to-speech that wouldn't touch the already-maxed-out VRAM. Piper (now maintained as OHF-Voice/piper1-gpl) is exactly that: a standalone neural TTS binary that runs entirely on the CPU, uses no GPU memory, and synthesizes far faster than real time. The en_US-amy-medium voice is a 61MB ONNX file.

echo "Hello! I am running on a GTX 970." | \
  ~/piper/piper/piper --model ~/piper/voices/en_US-amy-medium.onnx \
  --output_file /tmp/test.wav

On the Ryzen 5 3500X it synthesized an 11-second clip in 0.63s — a real-time factor around 0.05, roughly 18x faster than playback. Here's what came out:

The GPU does the thinking, the CPU does the talking, and the two never fight over memory.

The full loop: speech in, speech out, 2.8 seconds

Stitched together, it's a real assistant: a WAV goes in, Gemma transcribes and answers on the GPU, Piper voices the reply on the CPU, a WAV comes out.

[mic WAV] → Gemma 4 E2B (GPU, ~3.5GB) → transcribe + understand + answer
                                            │
                                            ▼
                                      Piper TTS (CPU, 0 VRAM)
                                            │
                                            ▼
                                       [speaker WAV]

I spoke "this is a test of speech recognition on a GTX 970 graphics card." The assistant heard it, understood it, and replied — that's the clip up at the top of this page.

Stage	Time	Where
Transcribe + understand + answer	1.9s	GPU (GTX 970)
Speech synthesis (TTS)	0.9s	CPU (R5 3500X)
End-to-end	2.8s	—

2.8 seconds, fully offline, on 2014 hardware. Stream the output (synthesize while the model is still generating) and you could shave that further.

What it cost

Nothing, in my case — the GTX 970 lives in a PC I'd already retired, so the whole build was $0. The only line item you'd actually hit reproducing it from scratch is the card itself:

Item	My cost
GTX 970 (retired PC, already owned)	$0
Ryzen 5 3500X + B450M + 16GB DDR4 (already owned)	$0
llama.cpp + Piper	$0
Gemma 4 + Piper voice	$0
Total	$0

And to be clear, the point isn't to go buy e-waste. A used 970 is about $15 (NT$500) if you wanted one — but the real idea is that the card already sitting in a retired PC, or in a drawer, might have one more job in it. This is what that second life can look like: an offline assistant that sees, hears, talks, and codes.

Takeaways

Where the time went

The VRAM headroom, not the speed. Getting a 3.2GB model and a 942MB projector to coexist in 3.5GB with 535MB to spare was the whole game — one more big component and it wouldn't have loaded. The image-ordering quirk also ate time until I found it: put the image after the text and the model swears there's no image.

Reusable diagnostics

When multimodal benchmarks faster than text, don't believe it at face value — check whether you're measuring prefill or decode. Fewer input tokens reliably speed up prompt processing; they don't reliably speed up generation. Any "this modality is faster" claim that doesn't separate the two phases is suspect.

The general principle

For this build, the GPU was never the bottleneck — VRAM headroom and whether the pieces composed were. A tiny multimodal model plus a CPU-only TTS engine turns a junk card into a complete assistant. The hard constraint was fitting everything into memory, not buying the hardware.

What a GTX 970 can actually do

Capability	Status	Speed
English / Chinese chat	✅	47.6 tok/s
Tool use (incl. parallel calls)	✅	47.6 tok/s
JSON / code generation	✅	47.6 tok/s
Image understanding	✅	~51 tok/s (prefill-flattered)
Speech recognition (EN / ZH)	✅ / ⚠️ mostly	~52 tok/s (prefill-flattered)
Speech synthesis (TTS)	✅ CPU, ~18x realtime	0.9s
Full voice assistant loop	✅	2.8s end-to-end

A 2014 card, a 2026 model, and a free CPU TTS engine — a complete offline multimodal voice assistant — and for me, $0, because the card was already sitting in a retired PC. You don't need a 5090 to get in the door, and you might not need to buy anything at all: that old card in a closet or a drawer could have a second life as an offline assistant like this one.

---

Appendix: the audio API

Audio goes in as an OpenAI-style input_audio content block, base64-encoded 16kHz mono WAV:

payload = {
    "messages": [{
        "role": "user",
        "content": [
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
            {"type": "text", "text": "Transcribe exactly what is said."},
        ],
    }],
}

• Modality order: Google recommends image-before-text and audio-after-text for best results. It's a recommendation, not a hard rule — but in my run, putting the image after the text made the model claim it saw nothing. (My audio clip went in before the text and still transcribed fine.)
• Audio limits: 30 seconds max, 16kHz mono WAV.
• base64 size: an audio clip's base64 blows past shell argument limits fast — send the request from Python, not curl.
• Built llama.cpp for the 970 with -DCMAKE_CUDA_ARCHITECTURES=52 (commit 42a0afd59, June 2026).