[Dev Workflow] Why an AI Agent's Memory Needs a Distilled Layer Above Search

Plain-Language Version: Finding a Note Again Isn't the Same as Reusing It

I write everything I learn into plain text files, one fact per file, and I gave my AI assistant a search box so it can find them (that was the last post). That works — until you notice search only ever hands back the raw note, or the raw code. The assistant still has to read it and work out what it means, from scratch, every single time. Finding the page again isn't the same as remembering the answer.

So I added one more layer: a small, curated set of files where each one is a finished conclusion, not raw material — "I tested this obvious-looking idea and it lost money, here's the proof." The assistant reads the conclusion instead of re-deriving it. This post is the design of that layer: what it is, why it's there, and the one thing people get wrong about it — its job is to stop me explaining the same thing over and over, not to force all my different AIs to think alike. The messy part, where a model ignored a note it had and guessed wrong, is at the end and clearly marked.

---

Preface

Part 12 ended on a loose thread. Even with search and a graph doing their jobs, retrieval hands you back raw material — the code, the raw note — and you still have to work out what it means, every single time. That's the tax this post is about.

Picture the difference between a mine and a workshop's parts bin. Search is a very good mine: ask for iron and it finds you the ore, fast. But you can't bolt ore onto anything — someone still has to smelt it into a part. If every job starts by re-smelting the same ore, the mine was never the bottleneck; the re-smelting is. A distilled memory layer is the parts bin: you smelt once, label the part, and every later job takes it off the shelf.

This is Part 13 of the AI Workflow series. Part 12 built retrieval — search plus a graph over the files. This one adds the layer above retrieval: a distilled, canonical record of conclusions, so the agent reuses the answer instead of re-deriving it. Design first — what the layer is and why it exists — then, separately at the end, the wrong turns.

The ceiling of retrieval: it returns the ore, not the part

The moment retrieval works well, a subtler cost surfaces. Search hands the agent the raw note or the code file — the material a conclusion was once smelted from, not the conclusion itself. So every new session that pulls up the same source pays the same cost again: read the 200-line file, reconstruct why it matters, arrive at the same decision it reached last week. Across a fleet of agents and a stream of fresh sessions, that's the identical expensive thinking, redone over and over.

Retrieval answered "can the agent find it?" It never touched "does the agent have to re-derive it every time?" Those are different problems, and the second one is where the cost actually lives. Finding is cheap; re-deriving is not.

What the distilled layer is: one file, one claim, with its evidence

The distilled layer is another folder of markdown, but under a stricter contract than the raw notes. One file holds one claim — a settled conclusion — with the evidence that backs it and what it implies. It isn't a dump of everything I've learned; it's the curated subset that earned the word canonical.

Things enter through a lifecycle, not a straight write. An untrusted intake lands in an inbox; a review step looks at it; only then does it get promoted to canonical. And there's exactly one promote button — mine. An agent can propose into the inbox; nothing becomes canonical without me pressing the button. Retrieval over this layer then hands the agent the claim, already reasoned, instead of the ore it came from. Everything downstream — every agent, every session — reads the same finished conclusion.

Why the goal is "stop re-explaining," not "make the agents agree"

Here's the objection that sounds obvious: if five different AIs read the same canonical notes, won't they still interpret them differently? Yes. And that's fine — because making them agree was never the goal.

The goal is narrower and far more useful: do the expensive thinking once, and never re-explain it. I run a small fleet — Claude Code, Codex, a few self-hosted models. The tax I actually pay isn't disagreement between them; it's re-explanation. Telling each new agent, and each fresh session of the same agent, the same handful of things: who I am, what I've already ruled out, which obvious-looking improvement I tested and threw away. Distillation kills that tax. The conclusion is written down once; every agent reads it instead of asking me.

So the design deliberately does not chase consistency:

• It tolerates drift. Two machines can hold slightly different states and I don't mind — eventually-consistent is fine. I don't ask "which copy is the real one."
• It keeps no AI-review gate. The flow is: I drop a source, an agent researches it, I decide, it's recorded. No agent reviews another agent into canonical.
• It keeps one promote button, mine. Agents propose into the inbox; promotion is a human call.

What makes tolerating drift safe isn't a consistency enforcer — it's a supersede-and-re-verify habit. When a new decision overrides an old note, the old note gets a banner and a log line, and recall re-checks a note against current state before trusting it. Staleness is caught at read time, not prevented at write time. That's the trade I'm making on purpose: I accept that two copies might disagree for a while, in exchange for never running a reconciliation machine.

The trap I'm avoiding is the factory: an elaborate consistency checker, an auto-reviewer, a cross-machine reconciler — each of which turns "the memory" into a new system I have to maintain. The lightweight version wins because the entire point was to remove work, not to add a layer of it. (Eventual consistency is a feature here, not a bug to engineer away.)

Mirror vs. distillate: 352 raw files feeding 59 canonical ones

It's easy to read the numbers wrong — I did. My raw mirror had 352 files; the distilled canonical layer had 59. The intuitive reading is "you're behind, you never migrated the rest." That's backwards. The raw mirror is the full record; the distillate is the curated conclusion sitting on top of it, and it consumes the mirror as its upstream. 352-feeding-59 isn't migration debt — it's a digest of a record. (Single source of truth is a layering property, not a headcount.)

I told the retirement half of this story in Part 11: I nearly deleted that "stale" mirror before realizing it was a live source still feeding the digest. Here I only need the direction of the arrow — raw feeds distilled, never the reverse — because that arrow is what lets the distillate be small without being lossy. Nothing is lost by curating; the record is still whole underneath.

What distillation buys you

Put it all together and distillation gives you four properties I'd want in any agent memory:

• The expensive thinking is done once. Read, reason, decide — that happens at promote time. Every later recall is a lookup, not a re-derivation.
• More agents, no more explaining. Add a sixth agent tomorrow and it reads the same canonical layer at zero new explanation cost to me. The saving compounds with fleet size.
• Drift is cheap, not dangerous. Because staleness is caught at read (supersede plus re-verify), I can tolerate eventual consistency instead of policing it. No reconciler, no scheduled diff.
• It stays lightweight. It's still just markdown with a promote gate — no reviewer bot, no consistency daemon, nothing that becomes its own maintenance burden.

That's the design. If you just want to apply it, the checklist at the end is the whole thing. If you want to see what it cost me to actually trust this layer, read on.

How I actually got here — the wrong turns

Below is what building and trusting this layer really looked like: one failure that taught me why grounding on canonical matters, and one result that finally convinced me the distillation was real signal, not luck. Neither is required to use the design — skip to the takeaways if you just want the shape.

A grounded model still reached for the textbook — the answer my own data had rejected

The scariest failure wasn't a missing note. It was an agent that had the note and didn't lean on it.

I asked an on-call agent about an internal operating rule I'd changed that morning — deliberately, because a rule that just changed is the one thing an agent can't have memorized. It can only be right by actually retrieving. The data was in place: the change had synced, search would surface it. The agent didn't look. It free-associated to the standard, textbook version of that rule — which happened to be exactly the practice my own tested evidence had rejected. A smart model with no grounding reverts to the textbook, and my whole edge lives in the places where the textbook is wrong here, proven by data.

That single incident is the entire argument for the distilled layer. The textbook answer is fluent, confident, and free — it falls straight out of the model's weights. The canonical note that says "the textbook is wrong here, and here's the evidence" is the thing that has to both exist and get consulted before the model fills the gap with its priors. Distillation gives you the first. The fix for the second was a retrieval-order rule: on any settled-doctrine question, consult canonical before your own past chat — because something you said yesterday isn't canon just because search can find it.

The proof it stuck: two disjoint distillates rebuilt the same picture of me

The result that convinced me came from the opposite direction. I spun up a fresh self-hosted model with a cold context and no access to my personal canonical memory at all. All it had were two distilled corpora that don't overlap: its own fleet-side memory, and the shared vault. From those two disjoint sources, with zero prompting from me, it reconstructed the same picture — who I am, what I've ruled out, the constraints I work under.

Two independent distillates, built from different raw material, converging on the same conclusions — that's the strongest evidence I have that the distillation is capturing real signal and not just echoing one lucky source. A single corpus can memorize noise; two disjoint corpora are far less likely to memorize the same noise. For that model, the re-explanation tax genuinely went to zero, and it got there by reading, not by asking me.

Takeaways

Where the time went. Not writing the canonical notes — trusting them. The hours went into the incident that proved a note can exist and still be ignored, and into building the retrieval-order and supersede habits that make a distilled note load-bearing instead of decorative.

Reusable diagnostics. Separate "found it" from "reused it" — retrieval that returns raw material still leaves the re-derivation cost unpaid. Test recall with a rule you changed today; a system that can only be right by retrieving is the only honest test of whether it retrieved. When an agent gives a fluent answer to a settled-doctrine question, check whether it retrieved or free-associated — the textbook answer is always available and often wrong for you. And read numbers like "352 vs 59" as layers, not as debt.

The general principle. The distilled layer exists to do the expensive thinking once and reuse the conclusion — its purpose is to kill re-explanation, not enforce agreement. Tolerate drift, keep yourself as the only promote gate, and catch staleness at read time with supersede-and-re-verify. The moment your memory grows a reviewer bot and a reconciler to force consistency, it has stopped removing work and started being work.

The distillation checklist

1. Distill conclusions, not just notes. One file, one claim, with its evidence and implications — something an agent can act on without re-reading the source.
2. Keep a lifecycle and one gate. Inbox for intake, a review step, promotion to canonical — and exactly one promote button, yours. Agents propose; you promote.
3. Optimize for "explain once," not "agree everywhere." Tolerate drift between machines; don't build a consistency enforcer.
4. Catch staleness at read, not write. Supersede an overridden note with a banner and a log line; re-verify a note against current state before trusting it.
5. Ground on canonical before priors. On settled-doctrine questions, consult the canonical layer before the model's own past chat — or it will fill the gap with the textbook.

Next: the persistent-knowledge axis is only half of it. The other half is live state — what an agent is doing right now — and how a fleet of AIs hands off work without re-explaining that either.