[Dev Workflow] AI Slop Is Structural: 93.2% Detectable With Every Style Tell Stripped

A good forger can match the brushstrokes. What gets him caught is the composition.

StoryScope (arXiv 2604.03136, COLM 2026) is that idea applied to fiction. The authors throw out every style signal — word choice, punctuation, sentence rhythm, all of it — and classify human versus AI on narrative structure alone. Macro-F1 93.2%.

A style-only classifier on the same corpus gets 85.8%.

Structure alone beats style alone, 93.2% against 85.8%

Features used	Macro-F1
Style only	85.8%
30 core structural features	84.8%
Full structure	93.2%
Structure + style	96.0%

Thirty structural features, with no style information at all, land within a point of the entire style-based approach. That is the finding.

The corpus: 10,272 human short stories from Books3, each paired with generations from five models — Claude Sonnet 4.6, GPT-5.4, Gemini 3 Flash, DeepSeek V3.2, Kimi K2.5 — for 61,608 stories total (24 short of a clean 5x: the models refused that many), mean length 4,753 words.

The surface tells are dying, and one fine-tune kills them outright

Em-dashes, "delve", the rule of three. All surface. GPT-5.4 already cut its em-dash usage, so half of what people are scanning for is a habit of one model version.

Two measured results, and neither is good news for a word-list gate:

Intervention	Effect on detection
Fine-tune on human fiction (Chakrabarty 2026)	AI-detection on creative writing 97% to 3%
LAMP style rewrite, against StoryScope	95.5% to 93.9%

One fine-tuning pass takes a 97% detector to 3%. The same class of attack aimed at StoryScope's structural features moves it from 95.5% to 93.9%. If your gate reads words, a single fine-tune retires it. If it reads structure, a rewrite barely registers.

Five questions that find AI drafts faster than any word list

These are the ones I actually use on a draft. Each pair compares AI-written stories with human-written ones across the full corpus.

1. Does the narrator state the theme outright?
2. Is emotion always rendered through the body?
3. Are there any subplots at all?
4. Are references named, or gestured at?
5. Is anything left unresolved?

#	Signal	AI	Human
1	Narrator states the theme outright	77%	52%
1	Thematic explicitness (1-5)	3.94	3.28
1	Dialogue used for philosophical debate	59%	34%
2	Embodied metaphor for emotion	81%	38%
2	Olfactory imagery	82%	57%
2	Emotion named outright	8%	29%
3	No subplots at all	79%	57%
3	Resolution driven by protagonist choice	69%	46%
3	Causal-chain continuity (1-5)	4.20	3.92
4	Names a real work or author	24%	47%
4	Vague allusion instead	72%	50%
5	Resolution as internal understanding	47%	27%
5	Morally ambivalent protagonist	38%	59%

The paper's word for the pattern is over-determination. Every element is doing declared work. The theme gets said out loud, emotion always lands in the chest or the throat, causality never slips, the protagonist chooses and the choice resolves the story.

Take the last two rows together. AI protagonists are morally clearer — ambivalence shows up in 38% of AI stories against 59% of human ones — and AI endings are nearly twice as likely to resolve into an internal understanding. A legible protagonist, then a lesson learned. That is over-determination arriving at the level of character.

That is the shape underneath all five questions. The model leaves no loose ends, and loose ends are the human fingerprint.

Switching models does not move you out of the AI cluster

The common reaction to slop is to try another model. The geometry says no.

Comparison	Distance
Human to AI centroid	6.6
Mean AI to AI	4.3
Closest human-AI pair	6.2
Two most distant AI models	6.0

The human-to-AI gap is 1.6x the mean AI-to-AI gap. And the second half of that table is the sharper version: the closest human-AI pair in the whole set is still farther apart than the two most distant models. The six most-confused source pairs are all AI-to-AI.

You can move between models. You cannot move out of the neighbourhood.

Every model has a structural personality, and Kimi's is not having one

Counting fingerprint features — structural traits that distinguish one source from the rest:

Source	Fingerprint features
Human	32
Claude Sonnet 4.6	26
GPT-5.4	11
Gemini 3 Flash	11
DeepSeek V3.2	7
Kimi K2.5	3

Model	What it does structurally
Claude	Restraint. Flattest event escalation, writes epilogues, avoids dream sequences, reverent and continuist toward literary tradition at 62% against 39-56% for the others
GPT	Gossip and rumour as a plot mechanism at 64% against 44-55%; distant retrospective narration
Gemini	Tidiest endings, longest denouements, bleakest settings — 88% tagged bleak
Kimi	Three fingerprints, lowest per-source F1 at 0.55, sits at the generic centre

Kimi is the interesting one. Having no personality is its fingerprint: it lands where the average AI story lands, which is itself a position.

Six-way authorship attribution on structure alone comes out at 68.4%. Not just human-versus-AI — which AI, from narrative shape.

I scanned my own published article and the disease was in the template

I ran the five questions over a piece I had already shipped: Fixing NCCL's stub-library error, 15,194 characters.

One hard hit. The same lesson appeared in four places.

Location	Section	Verdict
L32	Preface	Keep — it is the motivation
L236	"What this corrects"	Cut
L238	"What this corrects"	Cut — it restated L236
L304	Takeaways	Keep — it is the closing

The preface and the takeaways both earn their place; one sets up the problem and one closes it. The two in the middle were pure redundancy, and the second restated the first inside the same section.

After the edit the article came out 42 characters shorter, with zero facts lost. Forty-two characters is not the point. The point is that a reader hit the same sentence four times.

The root cause is the template, not the prose. I have a "what this corrects" section and a "takeaways" section. Those two headings are built to restate each other. No individual sentence was written badly; the structure requested the repetition and the model supplied it.

I have seen the section-level version of this before. In July I counted my back catalogue and found that 100 of 152 articles had exactly four FAQ pairs. Not four real questions — a slot filled to the brim.

So instead of tweaking a word list, I added the question to a review pass I already run. I added the question to the pre-publish review pass I already run, and I did not build the regex gate I originally wanted. The measurements behind that decision are below.

Deep dive

Skipping this section costs you nothing — the five questions and the template finding above are the usable part. What follows is the measurement record: what I built and threw away, and where I think the paper is thin.

Half the paper doesn't apply to technical writing

The paper finds humans favour non-linearity: anachrony 2.58 for humans against 2.31 for AI, delayed disclosure 1.96 against 1.68. Straightforward advice would be to withhold, reorder, let the reader assemble it.

That advice is wrong for this blog. A technical H2 has to be answer-first, both for readers scanning and for GEO, the search-answer equivalent of SEO. And I was corrected twice in July on exactly this: detective-style build-up reads worse than teaching-style for this material.

So the checklist I wrote from the paper has an explicit does not apply section. Half the findings are about fiction and stay there. Importing them wholesale would have made the writing worse in a way the source paper would have scored as more human.

Why I didn't build the regex gate

The obvious move after the self-scan is a regex. I prototyped one that catches restatement-marker openers — 換句話說 ("in other words") and its relatives — deliberately excluding my own transition habits so it would not just flag my voice.

Then I ran it over the corpus that matters, which is my own archive.

Measurement	Result
All 172 zh-TW articles	8 hits, 4%, median 1
Positive control, recall	1/3 — missed both markerless cases
Positive control, precision	1/2 — one hit was a real new conclusion, not a restatement

The positive control is the known-bad original, the article I had already found four repetitions in.

That leaves no usable threshold. At 2 it never fires. At 0 it false-flags articles I already published and still stand behind. I got the same result when I measured sentence rhythm: nothing there to catch, so I didn't build the check. A gate that fires on your back catalogue is not a gate; it's noise — the calibration lesson from the AI-tic check that never ran, arrived at from the other direction.

External review caught 3/3, including one I had missed

Instead of a regex, the question went into the review pass that already exists. Two external models read every draft for translation-ese; adding one more question to a prompt costs nothing, and dispatching to Codex is already routine here — see the 41-minute ticket.

Codex caught 3/3 of the repetitions I had found, with zero false positives. It also found one I had missed entirely — a three-step restatement chain inside a single argument:

L216: the bus was never saturated

L218: only 8.2% of Gen3 x16

L218: dropping 8.2% to 0.1% saves you a road that was already empty

Three tellings of one fact, the last two on the same line. I had read that passage many times and never registered it as repetition, because each sentence is doing rhetorical work — statement, evidence, implication. It is only repetition if you are looking for the shape.

It also applied my exemption correctly without being pushed, quoting it back:

a recap in a concluding section isn't automatically the disease

That is the exemption that keeps the gate usable. A model that can only pattern-match would have flagged the takeaways line at L304.

The paper's durability claim rests on one experiment, and that experiment is a tautology

This is where I think StoryScope is weakest.

The whole "structural features are durable under adversarial rewriting" argument rests on a single run: 278 Gemini stories pushed through LAMP, giving 95.5% to 93.9%.

LAMP is a style-rewriting system. By design it does not touch structure. So the finding is that an edit which does not change structure does not change structural features. That is a tautology dressed as a robustness result.

The real adversarial tests would be prompting explicitly for non-linearity, or fine-tuning on human fiction. Neither was run — even though the paper itself cites the fine-tuning result that took detection from 97% to 3%.

Every label rests on two annotators and 12 stories

Inter-annotator agreement is κ=0.74. Model-versus-annotator agreement is κ=0.84.

The humans agree with each other less than the model agrees with the humans. That was measured on 12 stories, 240 items, 2 annotators. Every label in the 61,608-story corpus rests on that validation.

I do not think it sinks the paper. The effect sizes are large enough to survive noisy labels. But "the model is more consistent than our annotators" is usually reported as a warning, not as validation.

The rarity tail is reported as raw counts, and that hides the ratio

Mean rarity percentile: 0.71 for humans, 0.49 for AI, Cohen's d 0.83. Humans are more original on average, and it is not a small effect.

Then the tail:

Slice	Human	AI
Rarest 10%	340	487
Top 1%	42	41

The paper calls that tail "mixed". But the test set holds 6,885 AI stories against 1,377 human ones — a five-to-one split printed on the same page as the Cohen's d. As rates, the rarest decile is 24.7% of human stories against 7.1% of AI ones, and the top 1% is 3.0% against 0.6%. Humans lead the tail by a wider margin than they lead the average.

What survives is the part that matters for copyright: 487 AI stories still made the rarest decile. Disputes judge individual works, never distributions, and a low rate is not an empty set.

I read this backwards the first time and wrote the inverted version into the draft. The pre-publish review caught it.

The irony: a 512-token baseline hits 99.9%

The paper's own baseline is a fine-tuned ModernBERT. It scores 99.9%.

ModernBERT has a 512-token input limit. On a corpus averaging 4,753 words, that is the first two or three paragraphs of a 5,000-word story.

So the AI tell is dense enough at the opening that none of the 304-question structural pipeline is needed to detect anything. The pipeline's value is not detection power — a small classifier reading three paragraphs already beats it. The value is the breakdown: 304 questions telling you what the difference is, feature by feature, which is the part you can act on when you are the one writing.

Detection was solved. Diagnosis is the contribution.

What I actually do now

Three things came out of this.

The five questions go into the draft review, because they catch what word lists can't: over-determination, no subplots, nothing unresolved. The regex did not get built, because measuring it on 172 of my own articles gave no threshold that fires on the disease without firing on healthy work. And the repetition question went to the external reviewer that was already reading every draft, where it caught 3/3 plus one I had missed.

The real surprise was the template. I went looking for bad sentences and found a heading pair that manufactures repetition on its own. If you run a house structure over every post, audit the structure before you audit the prose — the prose is downstream.

This article has three FAQ entries. That is the whole intervention.

---

Also in this series: Part 19 — From 41 minutes to 73 seconds · Part 17 — My skill had the check written perfectly, it just never ran

---

References

Jenna Russell, Rishanth Rajendhran, Chau Minh Pham, Mohit Iyyer (University of Maryland, College Park) and John Wieting (Google DeepMind), StoryScope: Investigating idiosyncrasies in AI fiction, COLM 2026 (arXiv:2604.03136).