[Dev Workflow] The AI-Tic Check I Never Ran — Because My Own Checklist Said 'By Hand'

In plain English: why a spell-checker for tone is hard

Here's the version for a friend who doesn't do this for a living.

When you have a language model help you write, the obvious failure is that it makes something up. That one is easy to check: you go look at the source.

The subtle failure is tone. The model writes fluent, correct sentences that nobody would flag individually, but it has habits. It reaches for the same rhetorical move over and over. Read one paragraph and it sounds thoughtful. Read a whole article of it and you start to feel like you're being performed at.

The habit that got me: every time the article had to admit something inconvenient, it first announced that it was about to be honest. "I'm not going to pretend otherwise." "I'd rather flag that than quietly fix it." Each one is a fine sentence. Five of them in one article is a personality, and not a good one.

I had reviewers checking that draft. They passed it, because they were looking for a different problem. And I had a checklist item that describes exactly this habit — but the item said to check it by hand, so in practice I never did.

---

The pipeline: paranoid about facts, blind to tone

I have a fairly paranoid publishing pipeline. Facts get verified against primary sources by one model; the prose gets read for translation-ese by two more, in both languages; a local model plays a mid-level reader and tells me which sections it lost the thread on.

Last night a draft went through all of it and came out clean. Then the one human in the loop read it and stopped on a single sentence.

This is what the gap turned out to be, and what I did about it.

The tell: one sentence, and then the same move four more times

The feedback was short: that sentence pattern has always felt weird to me. The sentence was:

So bandwidth is doing real work here, and I'm not going to pretend otherwise.

My first instinct was to fix that line. Then I searched the draft for the shape of it rather than the words, and found the same move four more times:

…and I'd rather flag that than quietly fix it.

Full disclosure on that comparison: the single-card figure is a 128-token generation…

I should be exact about what I did here. I read the source; I did not run that file.

One caveat I should state rather than bury.

Every one of these sits immediately before something unflattering — a measurement that isn't apples-to-apples, a conclusion I got wrong, a thing I inferred rather than observed. The admission is real in all five cases. What's added is the announcement: a short passage where the writer vouches for their own integrity before spending it.

I've started calling it performative candour. It's distinct from throat-clearing, which is a wind-up before any point at all, and from flattering the reader, which is a different kind of grease. This one flatters the writer. And it grows specifically in first-person debugging writing, where every section has something to admit — so the tic reproduces once per section until the whole piece is a character study.

The fix is not to delete the honesty. In the revision I kept every admission and cut every announcement:

Before	After
So bandwidth is doing real work here, and I'm not going to pretend otherwise.	So bandwidth is doing real work here.
Then I over-corrected, and I'd rather flag that than quietly fix it.	Then I over-corrected.
I should be exact about what I did here. I read the source; I did not run that file.	I read the source; I did not run that file.

Same facts, one fewer performance.

Two reviewers cleared it, because they were checking for the wrong disease

The part that bothered me more than the tic: this draft had passed two independent native-speaker reviews, in both languages, from two different models. One of them described the English as reading "impeccably".

They weren't wrong, and they weren't lazy. They were answering the question I gave them, which was: does this read like it was translated from Chinese?

It doesn't. Every one of those five sentences is fluent, idiomatic, native English. A reviewer hunting for translation artifacts has no reason to stop on "I'm not going to pretend otherwise" — it's the kind of thing a native writer says.

Translation-ese and AI-ness are different diseases. I'd built a good gate for the first one and quietly assumed it covered the second. It doesn't, and the overlap is smaller than it looks: the more fluent the model gets, the cleaner it passes a fluency check, and the more its remaining tells are structural rather than lexical.

The root cause: "do this by hand" is not a checklist item

My writing skill already had a section for this. It lists sentence-length rhythm, punchline density, inversion overuse, em-dash overuse — a decent inventory of exactly the patterns that were in the draft.

Its heading read: "rhythm + AI structural tells (manual sweep, grep can't catch these)".

So there was no command. No exit code. Nothing that would stop a commit. And when I look at what I actually ran on that article, I ran the mechanical anti-pattern grep, I ran the local reader-model comprehension pass, I fired the four external review calls — and I skipped this one. Not as a decision. It simply never became an action.

A rule that exists only as "remember to look at this" does not exist. It's a note about a rule.

I'd written down the principle for this the day before, and then failed to apply it. Freeze mature, repeated, machine-checkable judgments into code — otherwise a model reinterprets the instructions on every run and the criteria drift. This section was the textbook case, and I'd left it sitting in a document.

Calibrating against my own back catalogue, not an ideal

So I split the section in half. The countable patterns became a script; the genuinely human ones (does this paragraph carry information, is this a real objection or a strawman) stayed prose.

The interesting problem wasn't the regexes. It was the thresholds.

The temptation is to set them where you think good writing should sit. That produces a checker that fires on everything, which trains you to override it, which leaves you exactly where you started with an extra step. So I did the opposite: I ran the checker over five articles I had already published and was happy with, and read the distribution as house style.

Pattern	My 5 published articles	Threshold I set	Blocks?
Performative candour	0 of 5 had any	1	yes
"not X, but Y" inversions	1 of 5 reached 5	5	warn
Em dashes	5 of 5 ran 2.7–7.2 per 1k chars	7.5 per 1k	warn
Chained 的 (zh)	5 of 5 ran 1–6 sentences	6	warn

The row of zeros is the crux of the whole design. Performative candour appears nowhere in five articles I wrote before this one, and five times in the draft I'd just produced. That's not a stylistic preference I'm now enforcing against myself. That's drift, and drift is the only thing worth a hard block.

Everything else is house style with a long tail, so it warns and moves on.

My first threshold was wrong, and my own archive told me so

I set the em-dash limit at 3.0 per thousand characters. It was too strict by roughly a factor of two.

Four of the five published articles failed it.

That's not a gate. That's a smoke alarm in a kitchen. The house voice genuinely uses em dashes at roughly twice the rate I'd guessed, and if I'd shipped that threshold I'd have spent a month clicking past it until I stopped reading the output at all. I raised it to just above the observed ceiling, so it now only speaks up for a genuine outlier.

Then I had to do it a second time, for a reason worth knowing if you build one of these. Partway through I changed what the checker counts as body text (see below), which shrank the denominator and inflated every density measurement. The em-dash ceiling moved from 6.4 to 7.2 per thousand characters, and one of my five published articles quietly slipped over the line I'd just drawn. A threshold expressed per unit of text has to be recalibrated whenever you change what counts as text. I raised it again and re-ran the archive.

After that, all five published articles pass clean — no blocks, no warnings — and the new draft still got caught. That's the behaviour you want: silent on your own history, loud on the new thing.

It blocked my own draft, which is when it started counting

I wired it into the publish gate, the same one that already refuses to let me commit an over-long summary card. Then I tried to publish.

gate-pass BLOCKED: Check A.5 — keep the fact, cut the announcement.

The English version of the article still had two instances. I'd fixed the Chinese, reported to my reader that it was done, and moved on — and the gate caught that the twin was still carrying the disease. I'd also had a case-sensitivity bug in the English regex that let "Full disclosure" through, which I only found because the block made me go look.

A gate that has never stopped you is a gate you haven't tested.

It then stopped this article, too, and that one was its fault. Ten hits, all of them the passages above where I quote the tic in order to diagnose it. The checker was counting citations as confessions. A style checker that can't tell the difference is unusable for writing about style, so it now ignores blockquotes and quoted spans and scores only the author's own sentences.

The regression test matters more than the fix: I fed it the same five sentences with the quote marks removed, and it still blocks. Excluding citation is not the same as excluding evidence, and if I hadn't checked, I'd have shipped a checker that goes quiet the moment you put something in quotes.

What it still can't catch: a mistake dressed as an insight

After all of this was live, I wrote the social posts for that article and sent the English through the native-speaker review one more time. It came back with a flag I hadn't anticipated:

Clean, reproducible, and answering a question nobody had asked.

That's my line about a benchmark I'd misconfigured. The reviewer's read: this is a mistake packaged as a witty insight, which is its own kind of performance — the mistake gets to be charming instead of just being a mistake.

It's a sibling of the tic I'd just gated. But my regex looks for self-announcing keywords, and that sentence has none. It doesn't say it's being honest. It just polishes the failure until it shines. I replaced it with "Total user error."

Then the same reviewer read this article, and the result is the part worth staying for.

It found three more instances — in the piece you are reading, the one whose entire subject is this tic. In English: "Here is the embarrassing part" and "The honest limit". In the Chinese twin, 「誠實的收尾在這裡」 — "the honest ending goes here". It also flagged that I'd written my own bad threshold as "because that felt disciplined", which packages an error as evidence of a virtue — the same move as the social-post line, one paragraph after I'd finished explaining why that move is bad.

My checker scored all four of those zero. Every one of them is performative candour by the definition I wrote three sections ago, and not one contains a string my regex was looking for.

That is the real limit, and it's sharper than "regexes can't read". The tic is a shape and a regex matches strings. I had extracted five phrasings from one draft and mistaken them for the pattern. The gate I built catches the sentences I had already seen. It caught nothing I hadn't.

So I widened the pattern list with the four new forms, re-ran the five published articles to confirm they stay clean, and updated this section — which is the loop working as intended, just not the loop I thought I was building. The script isn't a detector of performative candour. It's a memory of every instance one of us has caught so far, and it needs a reader to keep feeding it.

The division of labour comes out inverted from what I expected. I assumed the script would do the catching and the human would handle the judgment calls. What actually happens is the human does the catching, and the script makes sure the same thing never gets past twice.

Takeaways

Where the time went. The real time sink wasn't writing the checker; that took an afternoon. It went into calibration, and specifically into being wrong about it once. My instinct for "what a reasonable threshold looks like" was off by roughly a factor of two against my own published writing, in the strict direction. If I'd trusted the instinct I'd have shipped a tool that cried wolf.

Reusable diagnostics. Go through your own checklists and find every item whose instruction is "review this manually" or "use your judgment here". Then check whether it has ever actually happened. Mine had a clean record of never running, and it wasn't the only one. For the ones that survive, split them: the countable half becomes a script with an exit code, the uncountable half stays prose. And when you set the numbers, run the tool over work you already shipped and liked — your archive is a better spec than your taste, because your taste is what produced the archive in the first place.

The general principle. A quality bar that lives only in prose is a preference. It becomes a standard at the moment something can refuse to proceed.

Conclusion

If you write with a model in the loop:

1. Your fluency gate and your AI-ness gate are not the same gate. Fluent output passes the first one by definition.
2. Search your drafts for repeated moves, not repeated words. The tell is the shape appearing once per section.
3. Keep the admission, cut the announcement. If a sentence tells the reader you're being honest, the next sentence was doing that already.
4. Freeze the countable half of every "check this by hand" rule into something with an exit code.
5. Calibrate on your own published work. A threshold that fails your back catalogue is noise.
6. Don't trust a gate until it has blocked you.

The Wikipedia editors' catalogue of AI writing signs is the best public inventory of these patterns I've found, and it's where a lot of my list started. Performative candour isn't on it yet. Neither, I suspect, are the ones that will show up in next year's models.

---

Also in this series: Part 16 — Agent memory self-poisoning · Part 15 — Your agent's skills are a context budget