[Agent 101 #7] Give your AI assistant eyes and ears: vision + voice for a text-only brain

The plain version: how a text AI "grows" eyes and ears

By posts #1–#6 you've installed the assistant, hooked it to your phone, and let it run tasks on its own. But so far it's still a "text-only" assistant — paste a screenshot or send a voice clip and it's blind and deaf.

This post fixes that: it makes the assistant see and hear.

The key is one mental shift. Your instinct is "so I need a model that can see images, right?" — nope. Vision-capable multimodal models are big and expensive, and your existing brain is genuinely good. The right approach is splitting the workload: keep the text brain as the brain, and bolt a small, dedicated "seeing" model next to it as the eyes. The image goes to the eyes first, becomes a sentence describing what's in it, and that sentence goes to the brain. The brain reads only text the whole time — it has no idea it's actually looking at a picture.

Ears work the same way: voice becomes text first, then gets handled as usual. And the ears are already built into Hermes — you probably never noticed.

---

Intro

The first six posts got the assistant up and running. From here it's the advanced track — the toys get a bit more advanced, but I'll still keep it understandable (this series has always been about "easy to follow", not "showing off").

First up, the one with the biggest payoff: perception. An assistant that can see and hear is a completely different experience from one that only reads. And the trick is surprisingly cheap — cheap enough that I squeezed the "eyes" onto an eleven-year-old graphics card with room to spare.

---

⚠️ Does this post apply to you? This is mainly for a text-only main brain (like my ds4):

• Vision: if your brain is already multimodal (like the ChatGPT you set up in #4), you don't need this vision side-car — just send it the image. The side-car is for giving eyes to a text-only brain that can't see.
• Voice: this one's universal. Hermes always transcribes voice with the built-in faster-whisper first (the agent loop runs on text), and it's free, built-in, and on by default — so just keep it.

The core idea: text brain + perception side-car, not a brain swap

Let's get the architecture straight first, so the config makes sense later.

Your assistant has a main brain (mine is ds4, a 284B local model, text-only). It reasons, decides, replies. It can't see images or hear sound — that's a built-in limitation, not a bug.

Rather than replace this perfectly good brain, you bolt a perception side-car next to it:

• Eyes = a small vision model (Gemma 4 E2B + a vision projector). One job: turn an image into a text description.
• Ears = a speech-to-text engine (faster-whisper). One job: turn sound into text.

Both image and sound are translated into text before they reach the main brain. The brain always works in its comfort zone (reading text), yet behaves as if it can see and hear. That's the whole point of the side-car: outsource perception, keep reasoning in the main brain.

Vision: use Hermes's built-in auxiliary.vision, don't write a plugin

Hermes ships auxiliary.vision — you just tell it "here's a vision-capable endpoint", and when a message with an image arrives it automatically sends the image over, gets a description, and splices it back into the conversation. The config looks roughly like this (in your config.yaml):

auxiliary:
  vision:
    provider: auto          # with base_url set below, auto routes to your custom vision endpoint
    base_url: http://<your-vision-endpoint>/v1
    api_key: ""             # usually empty for a local self-hosted endpoint
    model: gemma-4-E2B-QAT-Q4_0.gguf
    extra_body:
      max_tokens: 512       # too small and the description gets truncated to blank

That endpoint is just a small Gemma model with vision bolted on (pass an extra --mmproj projector file at startup and it can see) — any vision-capable, OpenAI-shaped server works. Restart and it's live.

⚠️ Hermes officially marks "pointing vision at a non-first-party endpoint (anything other than OpenRouter / Nous)" as experimental — it works, but after setup actually send a test image to confirm it connected; if it breaks, clear base_url and leave provider: auto to fall back to the default.

Why use the built-in auto-hook instead of writing a "vision skill" the agent calls itself? Because your main brain is text-only — it literally can't see the image, so asking it to decide "now I should call the vision tool" isn't reliable (it's guessing blind). Letting the gateway auto-hook the moment a message arrives is far steadier than letting a sightless agent decide. And — an unmaintained, self-written skill becomes tech debt; use the built-in when you can.

Real-world test: a text-only brain reads a "CAT 42 / risk HIGH" image

Enough architecture talk — let's test it with an image. I gave it a picture with CAT 42 and risk HIGH on it (one in black, one in red):

• the gateway auto-routed the image to that Gemma eye and got back a description
• ds4 (the text-only main brain) took the description, reasoned, and replied in Traditional Chinese
• the result: it correctly read 42, CAT, and risk HIGH, and it even identified which word was black and which was red
• end to end about 1 min 48 sec (eye ~12 sec + the 284B's already-slow decode)

A model that is "blind" by design accurately described an image it technically can't see. The magic is all in the side-car.

Hearing: your Hermes probably already listens, you just never noticed

Voice is even cheaper, because Hermes ships faster-whisper, on by default. You've probably never configured it — because you don't have to.

stt:
  enabled: true   # this is the default

The first time it's used, it auto-downloads the speech model; after that, send a voice clip and it transcribes to text and feeds it into the conversation as usual. I tested it by using macOS say to read a Chinese sentence → converting to wav → faster-whisper transcribed it in a second or so, language auto-detected as Chinese.

Don't worry about Simplified Chinese: the main brain outputs Traditional anyway

Here's something you might worry about: the two small middle models — Gemma describing, Whisper transcribing — might emit Simplified Chinese, and small models occasionally mishear a word or two.

But this barely affects what you see, because the model that actually replies is your main brain. The description and the transcript are just intermediate material handed to the main brain; once it digests them, it re-renders in its own Traditional Chinese. You can feed it a mix of Simplified and Traditional, and it always outputs clean Traditional. So "upgrade the middle model to a more accurate one" is an optional nicety, not a must.

Absurdly cheap: the eyes run on a 2014 graphics card

Last, the hardware — because this is the most fun part. Those "eyes" — Gemma + the vision projector — run on a 2014 GTX 970 (~US$15 used), and the same card is doing something else at the same time (compressing long conversations for the main brain).

The reason it fits is that this model is very memory-frugal: it only "remembers carefully" the most recent tokens and keeps just a rough sense of earlier ones, so even a very long conversation barely touches the card's VRAM — and a 4GB old card can hold both "compression + vision" jobs at once. (For why this works and how much it saves, I break it down in the GTX 970 series.)

The architectural split is simple: the "cheap but always-on" perception work goes to the soon-to-be-e-waste old card; the expensive reasoning stays with the main brain. You don't need a new machine to let your assistant see the world.

The biggest hurdle wasn't wiring vision — it was realizing I shouldn't write a vision tool

My first instinct was the natural one: write a vision tool, let the agent call it when it wants to see. But it hits a dead end — the main brain can't see the image, so it has no reliable signal to decide "now I should look", like asking a blind person to judge whether there's a photo on the table. Using a gateway auto-hook to intercept messages solves it entirely. This leads to a pattern you'll reuse everywhere: perception should happen before the agent starts thinking — don't leave it to a sightless agent to decide.

A few more takeaways worth keeping: to make a text model multimodal, first ask "can I bolt on a perception side-car" before "should I swap in a multimodal model" — the side-car is cheap, detachable, and doesn't touch your existing brain. Use what the framework ships (auxiliary.vision, built-in STT) instead of writing your own skill; an unmaintained one is just tech debt. Don't panic if a middle step emits Simplified or mishears a word — as long as the brain that replies is good, it re-renders clean; spend your effort on the main brain, not the middle layer.

Bottom line: multimodal doesn't have to mean one brain that sees and hears — it can be one brain that thinks plus a few small organs that perceive. Split perception from reasoning and you can do perception on the cheapest hardware and save the good model for where thinking actually happens — which is exactly why a 2014 card can be the eyes of a 2026 brain.

Conclusion

• Text assistant grows eyes: point Hermes auxiliary.vision at a --mmproj Gemma endpoint; the gateway auto-hooks, and the main brain never needs to understand images.
• Grows ears: faster-whisper is built in and on by default; send voice and it auto-transcribes.
• Simplified in the middle doesn't matter; the main brain re-renders in Traditional.
• The eyes can be cheap: SWA keeps 64K context off the VRAM, so a 4GB old card runs compression + vision at once.
• ⚠️ Honest caveat: a 4GB card has only ~142MB of VRAM left after loading — a desktop spike can OOM it, so stress-test for long-term production; voice transcription occasionally slips a word (doesn't affect the final Traditional output).

---

This series — AI Agent Life, from Zero:

• Part 1: AI assistant vs ChatGPT
• Part 2: What is an agent framework
• Part 3: The ChatGPT-brain + Hermes-body combo
• Part 4: Install Hermes Desktop
• Part 5: Connect Hermes to Telegram
• Part 6: Let your assistant run on its own
• Part 7: Give your AI assistant eyes and ears (this post)

The hardware side: Gemma 4 on a GTX 970 series — which card these eyes run on, and why it fits.