[Dev Workflow] Agent Memory Self-Poisoning: When an AI Agent Trusts Its Own Wrong Answers

The plain version: my agent's own wrong answer outranked the corrected truth

I have an AI agent that remembers things between sessions. One day it gave a wrong answer to a question. That wrong answer got saved into its long-term memory automatically, with no one checking it. From then on, every time I asked, the agent pulled up its own old wrong answer and treated it as trustworthy — even after I'd loaded the correct answer into the knowledge base. At one point it retrieved both the wrong answer and the right one in the same breath, and instead of picking the correction, it averaged them. The problem was never that it couldn't find the truth. It's that it treats its own past mistakes as evidence equal to the truth.

---

The setup: durable memory that auto-saves its own conclusions, no human in the loop

I run a small fleet of home agents. One of them — a "Hermes sib," one seat in that fleet — is configured with long-term memory that persists across sessions and writes to itself with no approval gate:

memory_enabled: true
write_approval: false     # no human in the loop
flush_min_turns: 6

That write_approval: false is the whole story in one line. When the agent reaches a conclusion, it saves it. Nobody signs off. The upside is real — the agent accumulates its own working knowledge without me hand-feeding every note. The downside is this post: a wrong conclusion gets saved just as eagerly as a right one, and once it's in durable memory it comes back in every future session.

This is the companion to Part 11, about how an old agent-memory system died. This one is about how a new one poisons itself.

The probe: ask it something that changed this morning — the one test you can't fake

I'd just upgraded my knowledge-base cross-machine sync from daily to minute-level. Before trusting it, I wanted to confirm the new pipeline actually connected end to end. So I asked the on-duty agent about an internal ops rule — a risk-control rule on one of my private systems — that had changed that morning.

That choice was deliberate. If the answer only just changed, the agent can't recite it from memory or training — the fact didn't exist yesterday. It has to actually retrieve the current version, or make something up. That's the one test you can't fake: ask something the system didn't know yesterday, and watch whether it looks it up or fabricates. It turned into a five-round pathology of agent memory.

R1: no data, no habit — it cited the old rule, and honestly admitted it didn't know it changed

First round, the sync was still on the old daily cadence, so the agent was working off a twelve-hour-old worldview. It cited the old rule. Stale, but honest — it didn't know the rule had changed, and crucially it didn't pretend to. No memory, no retrieval habit, an out-of-date picture of the world: I'll take an honest stale answer any day. R1 was fine.

R2: the new rule was retrievable, but it free-styled the textbook answer my system had already rejected

By the second round the new rule was in the knowledge base — qmd finds it reliably, with a hit rate of roughly 87%. But the agent didn't search. It free-styled an answer from general domain knowledge, and the answer it produced was the standard textbook approach — the exact approach my system had tested empirically and rejected.

That's the insight worth keeping. An ungrounded smart model regresses to the textbook. But my whole edge here is built on the empirical fact that the textbook is wrong for this specific case. So the smarter the model, the more confidently wrong it is when it answers without grounding — it reconstructs the consensus answer fluently and never doubts it. R2 wasn't a knowledge gap. It was a missing retrieval reflex.

R3: I put a retrieval rule in its persona, and a clean session passed

So I added a retrieval rule to the agent's SOUL — its persona config. In plain terms: search first, read the source down to the line number, scan the "superseded / retraction" log before citing anything, then answer. I opened a clean session, asked again, and it did all of that and got the rule right. I thought I was done. I wasn't.

R4: it retrieved both the poison and the antidote, then split the difference

Here's the turning point. I asked the same question again — but this time on an agent instance that had already accumulated memory from the earlier bad rounds. Its answer opened, roughly:

Checked recent conversations: already defined (the old way)… also found the latest conclusion: (the new rule).

Read that carefully. It retrieved both — the poison (its own earlier wrong answer, now living in memory) and the antidote (the correct new rule from the knowledge base). And then it did the worst possible thing: it split the difference. It verbally acknowledged the new rule, but its actual conclusion still followed the old way. It didn't pick a side. It weighed a corrected error equally against the current truth and blended them.

That reframed the entire bug for me. The problem was never whether the agent could retrieve — it retrieved both, cleanly. The problem is it doesn't know which to trust, and its default is to treat its own past words as evidence on par with canonical knowledge. The error wasn't missed. It was vindicated — retrieved, weighed, and honored.

Why the poison sticks: approval-free auto-save, /new can't kill it, and it searches itself first

Three things have to line up to poison the memory this way, and all three were present:

1. Approval-free auto-memory. write_approval: false means a wrong answer auto-persists and then gets injected into the context of every new session. There's no gate between "the model said it" and "the fleet remembers it as true."
2. /new can't kill it. My reflex was to clear the conversation and start fresh. But /new clears the current conversation, not durable memory. I tested it: after /new, the same wrong answer came back verbatim. The hallucination had already moved into the long-term layer — a layer that command doesn't touch.
3. Its retrieval reflex is to search itself first. The live Telegram view shows the agent's first action verbatim as 🔍 Searching past sessions. It reaches for its own history before it reaches for canonical knowledge. The saving grace: session_search is a model-chosen tool, not a pipeline-forced step — which means the persona config can retrain the habit. That's the lever the fix pulls on.

The fix: three cuts, no model swap — this is a priority problem, not an intelligence one

Nothing here is solved by a bigger model. A smarter model retrieves the same two answers and faces the same choice. This is a priority problem, not an intelligence one, so the fix is three cuts to how the agent ranks and orders what it already retrieves:

1. Source-ranking rule into the persona. Canonical knowledge outranks recent conversation and outranks your own prior conclusions. When sources conflict, say it plainly — "my earlier statement was corrected" — and do not split the difference. Averaging a correction with the error it corrected is the actual bug; ban it explicitly.
2. Memory counter-injection. Write the "this is superseded" retraction into the same memory layer where the poison lives. Now when the agent retrieves the bad answer, it necessarily retrieves the retraction next to it. Send the cure down the same path the poison travels.
3. Tool order. For doctrine questions, query the canonical KB (musubi / qmd) first and session_search last. Reverse the reflex.

One line to remember: your own past words don't become a rule just because they got retrieved.

R5: same question, and it opens by superseding its own earlier answer

Same probe, one more time. This round the agent opened with:

Superseding what I said earlier…

It owned the mistake, cited canonical knowledge, got the reasoning right, and correctly rejected the old way. Full marks. The difference between R4 and R5 wasn't more memory or a better model — it was a ranking rule and a retraction planted where the poison would find it.

A different failure mode: the persona spoke before the evidence arrived

While I was at this, a different seat handed me a second lesson. I asked it about an internal metric threshold, and it misread the whole question as a software canary rollout — producing a tidy package of rollback drill, chaos test, and staged-rollout risk analysis, none of which touched what I'd asked.

Two causes, and the second is the important one:

• The word was ambiguous. "Deployment" means one thing in software and another in my internal context. The model picked the sense it already knew how to talk about.
• The persona fired before retrieval. Its structured persona template — strengths / risks / failure-modes / what-I'd-do — ran before any evidence arrived. The persona spoke first, so a well-formatted template filled an empty answer. It sounded like an assessment. It was scaffolding around nothing.

The fix rhymes with the main one: disambiguate-then-answer for ambiguous terms, and make the persona yield to evidence. The first line of any assessment now has to be Source: <x> or Not retrieved (reason). An ungrounded template output is just well-formatted hallucination — and a confident structure makes it more dangerous, not less.

Takeaways

Where the time went. Almost all of it went into telling apart two failures that look identical from the outside: "it didn't search" (R2) versus "it searched but didn't trust what it found" (R4). Same symptom — wrong answer — completely different disease. A missing-retrieval fix (add a search rule) does nothing for a trust/ranking problem, and I burned real time applying the R2 cure to the R4 disease before I saw they were different. One gotcha cost me a chunk of that time: HERMES_PROFILE=<profile> hermes chat silently falls back to default behavior, giving you false negatives in regression tests — you think a fix passed when you tested the wrong profile. Use the hermes profile alias wrapper instead.

Reusable diagnostics. Use a "changed-this-morning rule" as your probe. It's the only unfakeable test of any memory or retrieval system: because the answer just changed, the model can't recite it from training or from stale memory — it has to retrieve the current value or fabricate one, and you get to watch which. Every faked-retrieval bug in this post was caught by that single probe.

The general principle. Rank memory; don't pile up more of it. More memory without ranking just makes the poison stronger — every wrong answer you accumulate is one more retrievable "fact" arguing against the truth. The lever isn't volume, it's the rule: your own past words don't outrank canonical knowledge, and when sources conflict you own the error instead of splitting the difference.

The open question I'm not resolving: should auto-memory be approval-gated?

This one isn't fully settled. write_approval: false is the root enabler of the whole pathology, so the obvious move is to gate it — require approval before anything auto-saves. Gating stops wrong answers from persisting. But it also kills autonomous accumulation: if I approve every line, the agent isn't building its own memory — I'm just writing notes by hand with extra steps.

I chose to keep auto-save and lean on the ranking rule plus counter-injection — because the ranking rule is write-once, permanent, while approval is a daily chore I'll eventually stop doing. A one-time structural fix beats a recurring manual one. That's my call, not a proof; I'm laying it out so you can disagree.

What unsettles me isn't the bug. It's what the bug resembles: a machine that files its own mistakes as memory, retrieves them tomorrow, and believes itself — day after day, each recollection reinforcing the last. I've spent three years fighting staleness in a shared knowledge base. The same disease grew right back, quietly, inside one agent's private memory.

---

Also in this series:

• Part 9 — A Live-State Handoff Protocol for Claude Code and Codex
• Part 11 — Retiring an AI Agent's Memory System
• Part 13 — Why an AI Agent's Memory Needs a Distilled Layer