[Dev Workflow] From Markdown Search to a Knowledge Graph: How My AI's Memory Grew a Second Layer

Plain-Language Version: My AI Took Notes, Then Couldn't Find Them

I write everything I learn into plain text files — one fact per file, a few hundred of them. My AI assistant reads from that pile so it doesn't ask me the same thing twice.

To let it use the pile, I gave it a search box. That helped, until I noticed the catch: a search box only finds notes that use the same words as your question. Two notes about the same underlying problem, described differently, never show up together — even though they belong together.

A "knowledge graph" fixes that. It's a map, drawn automatically, of which notes are about related things — connected by ideas, not by matching words. This is the story of adding that map to my AI's memory, the two times I built it wrong first, and the odd way it later started tripping over itself.

---

Preface

Think of it like a workshop. First you need shelves so nothing's on the floor — that's storage. Then you need to find a specific tool fast — that's search. But the thing that actually makes a workshop feel like yours is knowing that the jig on shelf three and the bit in drawer nine are for the same job, even though nothing labels them that way. That last part isn't storage and it isn't search. It's a map of relationships, and you build it in your head over years.

Part 6 of this series handed you the tools — a search engine and a graph — as a way to stop burning tokens. This post is the layer underneath that one: how those two pieces actually came to exist in my setup, and why I added each one only after the layer below it hit a wall. I'll tell it in the order it happened — garbage versions and all.

Wall 1: ~600 notes my AI kept forgetting it had written

Here's the problem you hit the moment your agent has a memory: it rediscovers things it already knows. Different session, clean context, and it re-derives the same fix it wrote down three weeks ago — because a folder of markdown isn't retrieval, it's just a folder.

The first layer fixes exactly that. qmd is a local search engine over markdown — keyword in ~30ms, vector (meaning-based) in a couple of seconds. I won't re-explain the mechanics; Part 6 covers the three search modes and the token math. What matters here is that once it was wired in as a first-class tool, "query before you work" became automatic — one ordinary day logged 64 lookups against it without me thinking about it. The pile became memory.

And then the pile lied to me about how well it was working.

An audit said 82% of my searches were failing. It was wrong.

Picture opening a dashboard for your own memory and reading that 82% of your queries score "low." The obvious reading — the one I had — is that my knowledge base was full of holes, and I should go write the missing notes.

That reading was wrong, and the way it was wrong is the whole point. Keyword scoring necessarily comes up near-empty on a long, natural-language question, because you're asking in a sentence and it's matching tokens. The same queries that scored low on keyword scored around 0.7 on vector search. When I actually checked, only two notes were genuinely missing. The 82% wasn't a knowledge gap — it was an artifact of which retrieval method I was measuring.

The fix was small and it now lives as a hard rule: an empty keyword result, or a query longer than about four words, auto-retries on vector before doing anything else. But the interesting part isn't the fix. It's that vector search patched the "different words for the same query" problem and still left a bigger one untouched: it could not tell me what else related to a note I was already looking at. Ask "find the note about X" and meaning-matching works. Ask "what connects to X that I've forgotten about" and there's nothing to match against. That's not a search problem. That's the second wall.

Wall 2: search finds notes; it can't tell you what connects them

The situation that exposes it: you've got a note about "Ollama keeping models resident in memory" and another about "vLLM crashing on startup." They're the same problem — both are about one 128GB pool getting overcommitted — but they share almost no vocabulary. Search either one and the other never surfaces. The connection is real; it's just conceptual, and nothing in a search index encodes it.

What encodes it is a graph: notes as nodes, an edge whenever two notes are about the same concept, computed once and traversable. Then "what's adjacent to this?" becomes a real query, and so does "which notes are isolated, hot, or going stale?" — questions a search box fundamentally can't answer. The inspiration was open work in this space — Graphify's extract → graph → cluster pipeline, LightRAG, and an essay called "Stop Calling It Memory" that argues flat-file memory can't filter or traverse, so it isn't really memory yet.

The one decision I'm still glad about: I didn't move anything into Obsidian. The markdown files stayed the source of truth; the graph became a layer on top that never edits them. Files first, graph as a rebuildable derivative — that split is what let everything after this stay reversible.

I built the graph wrong twice in one afternoon before it worked

You'd think the hard part is building the graph. The hard part is building it wrong fast enough to learn what a good edge even is. I got three versions in one afternoon.

v1 — edges from embedding similarity. I connected notes whose text embeddings were close. The result was a hairball: my .py, .swift, and .tsx files dominated the graph, because two Python files are naturally similar as text whether or not they're related as knowledge. __init__.py came out as a hub. To see any structure at all I had to crank the similarity threshold to 0.90 — a tell that embedding distance is a bad edge metric here. 602 nodes, 142K edges, no signal.

v2 — markdown only, edges from shared concepts. I dropped everything but the markdown — about 291 files at that point — and drew edges from concept co-occurrence: two notes linked if they mention the same concept from a dictionary. Cleaner, but now 102 notes (35%) were isolated islands, because my concept dictionary was too narrow to catch what they were about.

v3 — wider concepts, weighted, with a fallback. I tripled the concept dictionary, added path-derived concepts, and let embedding similarity act only as a fallback where concepts came up empty. That did it: zero islands, ~7,000 edges, five real clusters. And the clusters told me something my folders didn't — my collection tags did not match the actual structure of the knowledge. Trading notes I'd filed together scattered across three different clusters. One concept, api, showed up in 172 of those 291 notes: a "god concept" so generic it connected everything to everything, exactly like a stopword in plain NLP. It had to be dropped for the rest to mean anything.

That afternoon is the part that never made it into the earlier, tidier post. The tool works now; getting there was three wrong graphs and a stopword.

I built a better tool and then didn't use it

Here's the part that annoyed me most. I finished the graph — I called it musubi, Japanese for "to tie together" — and for the "what's related to this?" question it was plainly better than plain search. Then I watched myself keep reaching for the old search box inside Claude Code and barely touching the new thing.

The reason had nothing to do with quality. qmd was exposed to the agent as a first-class tool it could see in its schema; musubi was "just a CLI." An agent reaches for what's listed in its tool schema, not for what's in my shell history — it doesn't know a command exists unless the interface tells it. The tool was more useful and less discoverable, and discoverability won every time.

The fix was ~150 lines: wrap musubi's search / neighbors / cold as an MCP server so the agent sees them as real tools. After that the division of labor was obvious: "find past experience, show me what's related" routes to the graph; "get me that specific file" routes to search. The lesson generalizes past this one tool: the last step of building something for an agent isn't making it good, it's making it visible. A tool the model can't see is a tool it won't use.

The graph bloated into a hairball, and my own index hijacked the results

Months of daily notes later, the live graph had ballooned to ~1,272 notes across every collection I index and over 200,000 edges — average degree in the hundreds. A hairball again, but for the opposite reason: too much real connection, not too little.

The specific failure was almost funny. I'd written an orientation doc — a hand-maintained index meant to help agents find their way around the knowledge base. That doc became the single biggest hub in the graph, with a degree over 1,100, because it references everything. So a search for one specific note started returning the index above the note itself: graph boost lifted the meta-doc over the answer it was supposed to point to. The thing I built to describe the system had become the loudest node in it.

The fix was stricter ranking: direct hits sort ahead of graph neighbors, meta-docs get downranked unless you name them, the neighbor boost is capped, and mirrored duplicates collapse to one canonical copy — 1,272 raw nodes down to 993. If that failure mode sounds familiar from Part 11 — a system's own self-description drifting away from what it's doing — that's the same shape, one layer down. A graph over your knowledge needs governance as much as the knowledge does.

Takeaways

Where the time went. Not building the graph — building it twice wrong. The whole first afternoon was really one discovery: embedding similarity is a terrible edge metric for a mixed corpus of code and prose, because code is self-similar regardless of meaning. The signal that worked was concept co-occurrence with the too-common concepts weighted down — which is just IDF, rare concepts carrying more weight than frequent ones. I arrived at it the hard way.

Reusable diagnostics. When a metric looks like a knowledge gap (82% "low"), check whether it's an artifact of the method before you act on it. Stopword your "god concepts" — any concept in more than half your notes is connecting noise, not signal. Ship the MCP server: a tool the agent can't see in its schema won't get used, however good it is. And watch your meta-docs — an index over the graph can quietly become the graph's loudest hub and outrank the answers.

The general principle. The layers are additive, not replacements. Files are the source of truth; search and graph are rebuildable caches stacked on top, and the arrows only point up — caches rebuild from files, never the other way. I added each layer only when the one below it hit its limit: a folder isn't retrieval, so I added search; search finds notes but can't traverse between them, so I added a graph. There's a third wall waiting — even good retrieval hands you back raw material, and re-deriving what it means every time is its own tax — but that's the next post.

Conclusion

If you're giving an AI agent a memory made of markdown, the order that worked for me:

1. Files first. One fact per file, plain text. This is the only layer that's the source of truth; everything else rebuilds from it.
2. Add search, and know its ceiling. Keyword for exact terms, vector for meaning. When a query is a long sentence, fall back to vector automatically — and don't trust a keyword-based "coverage" number.
3. Add a graph when you need traversal, not just lookup. Build edges from shared concepts, not embedding similarity, and stopword the concepts that are in everything.
4. Make it visible. If an agent is the user, the tool isn't done until it's an MCP tool.
5. Govern the graph. Rank direct hits over neighbors, downrank the maps you drew of it, and dedupe mirrors — or your own index will drown the answers.

Both qmd and musubi are open source, run locally, and work on any folder of markdown. Next in this series: why searching your notes still isn't enough — and the distilled layer I put above both.