[AI Workflow] Teaching a Chinese TTS Taiwan Mandarin: a 3-to-80-Entry Lexicon Plus g2pW Auto-Tagging

---

Preface

I use VoxCPM2, an open-source TTS model running on one RTX 2080 Ti, to narrate daily AI-news shorts in Mandarin with a clone of my own voice. The previous post covered the setup and the whisper check that transcribes every line and compares it against the script.

Since then, the failures changed shape. English names are mostly fine now. What breaks is ordinary Chinese characters. In 模型庫 ("model library") the last syllable should be kù, but the model said kū, which is 哭, "cry". In 門檻, "threshold", the second character went silent.

The cause is regional. VoxCPM2 reads Mandarin the way mainland China does, and my scripts are written for Taiwan, where some common characters are read differently. Any TTS trained mostly on one regional variant will make the same mistakes on another region's text. The model does not know which region you mean, so the fix has to be in the text you hand it.

My first approach was to add rules one at a time. The same errors kept coming back. A fix for 步驟 ("step") went in on October 13, and on October 19 the word was still misread. This post is what replaced it.

Put every fix in one shared lexicon that every narration path reads

The fix is one JSON file. Each entry maps a piece of source text to the text that actually gets sent to the TTS. Every narration entry point reads the same file: news shorts, explainer videos, single-line re-records. Subtitles always use the original script, so viewers never see the respellings.

The shared part matters more than the format. A fix goes in once, and every path picks it up on its next line.

The file grew fast once there was one place to put things:

Date (2026)	Entries
October 19	3
October 27	33
November 1	38
November 3, early morning	42
November 3, evening	80

The file looks like this. All of these are real entries:

{
  "entries": {
    "庫": "酷",
    "企業": "氣業",
    "檻": "砍",
    "步驟": "步揍",
    "發難": "發{nan4}",
    "帳本": "{zhang4}本",
    "曾侯乙": "{zeng1}侯乙",
    "研究": "研{jiu4}",
    "嗆你": "{qiang4}你",
    "被嗆": "被{qiang4}",
    "當機": "蕩機",
    "倒鉤": "到勾",
    "橋": "喬",
    "NVIDIA": "Nvidia",
    "AI": "A I"
  }
}

The lexicon holds three kinds of entries and nothing else

Fixes confirmed by ear. These are either homophone swaps or a pinyin tag on just the wrong character. 庫 ("storehouse", kù) became 酷 ("cool", also kù), which is how 模型哭 went back to sounding like 模型庫. 企業 ("enterprise") became 氣業, because Taiwan reads the first character qì, not the mainland qǐ. For 發難 ("to start trouble") and 帳本 ("ledger"), only the misread character gets swapped for pinyin: 發{nan4}, {zhang4}本. The rest of the word stays as characters.

VoxCPM's official phoneme tags. The VoxCPM usage guide defines tone-numbered pinyin in braces for Mandarin, like {xun4}{lian4}, and CMU phonemes for English, like {AH0 P AE1 CH IY0}. A tag replaces the character. It is not an annotation next to it.

Numbers spelled out. 2.0 goes to the TTS as 二點零, "two point zero".

Names and numbers are not added ahead of time. For a new name, I send the raw text first, re-roll seeds if it comes out wrong, and only tag it if it is still wrong after that.

Bind polyphones to words, not to single characters

A character with two readings must never get a single-character key. The key has to be the word.

曾 is the clearest case. In the name 曾侯乙 (Marquis Yi of Zeng, an ancient ruler) it is zēng. In 曾經 ("once, ever") it is céng. The entry is 曾侯乙 → {zeng1}侯乙. A key on 曾 alone would break every 曾經 in every future script.

嗆 ("to choke" or "to mouth off") has two readings, qiāng and qiàng. The lexicon has five phrase entries for it (嗆你, 被嗆, 嗆人, 嗆聲, 嗆回去) and no single-character key.

There is one deliberate exception. The early entry 檻 → 砍 is a single-character key, and I kept it. The Ministry of Education dictionary lists a second reading for 檻, jiàn ("railing"), but that sense rarely appears in news. A word-level key would be cleaner.

Matching is longest key first, and a matched span gets locked so shorter keys cannot rewrite inside it. This is the matching function:

def lexicon_spans(text):
    taken = [None] * len(text)
    for k in sorted(LEX, key=len, reverse=True):
        start = text.find(k)
        while start != -1:
            if all(t is None for t in taken[start:start + len(k)]):
                for j in range(start, start + len(k)): taken[j] = (k, start)
            start = text.find(k, start + 1)
    return taken

LEX is the entries object from the JSON file. Any position marked in taken belongs to a lexicon entry, and nothing downstream touches it. That includes the automatic tagger, covered two sections down. The lexicon always overrides automatic tagging.

When you are unsure of a reading, look it up in the Taiwan dictionary on moedict

My ear is not the authority on Taiwan readings. The Taiwan Ministry of Education dictionary is. 萌典 (moedict) serves it as JSON:

curl -s "https://www.moedict.tw/uni/曾"

The response lists each reading of the character as a separate heteronym. For 曾 there are two, shown in Zhuyin (the phonetic alphabet Taiwan teaches) and in pinyin: ㄗㄥ zēng and ㄘㄥˊ céng, each with its own definitions, so you can see which sense goes with which reading.

A reading from this dictionary can go straight into the lexicon. It does not need the ear test that a homophone swap needs, because the question it answers ("what is the correct reading here?") is a fact, not a judgment about how one TTS model renders a substitute.

Let g2pW tag what the lexicon misses, only where it disagrees with pypinyin

The lexicon only knows what I have already caught. For everything else, a second step tags characters automatically before the line goes to the TTS.

It uses two tools that disagree in a useful way:

• g2pW reads a whole sentence and predicts each character's reading in context, trained on Taiwan data. The model is about 606 MiB and runs on CPU.
• pypinyin looks words up in its phrase dictionary, which is in Simplified Chinese, so the text is converted to Simplified first. Its answer is the mainland reading. I use it as a stand-in for what the TTS might guess.

Where they agree, the TTS will probably get it right on its own, so the character stays as is. Where they differ, the character is replaced with g2pW's pinyin, for example {luo4}. Tagging only the disagreements matters, and the punctuation section below explains why tagging common characters backfires.

Three things are skipped even when the readings differ:

• 一 (yī, "one") and 不 (bù, "not"), because their tone changes with the next syllable (tone sandhi) and the TTS already handles that.
• Any neutral-tone reading, the ones that end in 5 in tone-numbered pinyin.
• 括 ("include") and 和 ("and"). The dictionary reads them guā and hàn, while everyday Taiwan speech mostly says kuò and hé. Following the dictionary there would sound wrong to Taiwan listeners.

The core of the tagging loop, after lexicon_spans has run. If you do not write code, skip it; it does exactly the comparison above:

ch, r = text[i], reading[i]      # r: g2pW reading in context
d = base[i]                      # d: pypinyin reading of the Simplified text
if r and ch not in SANDHI | COLLOQUIAL and d and not r.endswith('5') and r != d:
    say.append('{' + r + '}')    # tag it
else:
    say.append(ch)               # leave it to the TTS

If converting to Simplified changes the length of the line, the two readings no longer line up character by character. In that case nothing gets tagged. A wrong tag is worse than no tag.

My tagging script has a --show mode that prints only the tags it would add:

echo '["曾侯乙編鐘出土時，我曾經在現場。","他把垃圾丟進垃圾桶，研究人員看著樹葉落下。"]' | ./g2pw_tag.py --show

2 垃:le4(預設la1) 圾:se4(預設ji1) 垃:le4(預設la1) 圾:se4(預設ji1) 落:luo4(預設la4) | 他把{le4}{se4}丟進{le4}{se4}桶，研{jiu4}人員看著樹葉{luo4}下。

(預設 means "default", the pypinyin reading.) The first sentence, about the Marquis Yi of Zeng bells, contained both 曾侯乙 and 曾經 ("once") and printed nothing. The lexicon had already handled 曾侯乙, and for 曾經 both tools said céng. The second sentence printed three things. 垃圾 ("garbage") was tagged le4 se4 twice, where the default would be la1 ji1; Taiwan says lèsè, the mainland says lājī. 落 ("fall") was tagged luo4 instead of la4. The 研{jiu4} in that sentence came from the lexicon, not from g2pW.

On November 3 I checked g2pW on a set of known Taiwan readings. It got all of them right: 曾侯乙 with zēng, 落下 with luò, 垃圾 as lèsè, 薄 as bó, 微 as wéi, 研究 with jiù. pypinyin gave the mainland reading for 曾 and 落, so those are the two characters that get tagged.

When a tag fails, try a homophone or the raw character, and let your ears pick

A correct pinyin tag does not guarantee a correct reading. Sometimes the model simply renders the tag badly.

• 當機 ("to crash", for a computer) tagged {dang4}{ji1} was still wrong. The homophone 蕩機 worked.
• 倒鉤 ("barb") failed with pinyin. I picked 到勾, a different pair of characters with the same sound.
• 炸 ("to fry") tagged {zha2} came out as 抓 (zhuā, "to grab") in 4 of 4 takes. The raw 炸 with no tag was correct on the first take.
• 橋 ("bridge"): in a blind test the homophone 喬 sounded best.

So for any word the model still mispronounces, I render three versions of the line: the pinyin tag, a homophone, and the raw text. They are shuffled and labelled A, B and C, so I do not know which is which while listening. I package them as mp4 cards because my phone will not play wav. Then I pick by ear. From request to a new lexicon entry plus a one-line re-record takes about 30 seconds.

Whatever wins goes into the lexicon, and every pipeline gets it from then on.

Use full-width punctuation and keep tags off common characters

Two small habits in the text prevent a lot of strange output.

First, use full-width Chinese punctuation (， and ？), not the half-width ASCII , and ?. One outro line had half-width punctuation plus tags on two very common characters, {hai2} and {wei4}. It came out as gibberish; whisper transcribed it as 別在得去安的平安系, which means nothing. With full-width punctuation and the tags removed, it was correct on the first try.

Second, do not tag common characters. The model already reads them correctly, and a tag only adds a chance to fail. This is also why the g2pW step tags only where the two readings disagree.

The same caution applies when whisper misses a line. Converting the whole sentence to pinyin seems like a fix. I tried it once and the voice sounded distant. My limit now is at most three words of added pinyin in any one sentence.

English words in Mandarin scripts: lowercase env vars, keep letter-by-letter acronyms, and never put English first

Scripts about AI tools mention code, so English shows up in Mandarin lines. Three rules cover it.

Lowercase all-caps identifiers. Environment variables and flags in all caps get spelled out letter by letter or mangled. PARALLEL was transcribed as "PARLL". I send them lowercase with underscores turned into spaces:

ROUND_ROWS  →  round rows
DRY_RUN     →  dry run
CUDA        →  cuda

Keep acronyms that are read letter by letter uppercase. GPU and TTFT are read as letters anyway, so they stay as they are. The lexicon also has NVIDIA → Nvidia, so it is read as a word, and AI → A I, so the two letters are read separately.

Never start a Mandarin sentence with an English word. "overlay 要先登入…" ("overlay needs you to log in first…") was misheard 10 out of 10 times, as things like "Orly" and "AURAL". "這個 overlay 要先登入…" ("this overlay needs…") was heard correctly 4 out of 4 times. One Chinese word in front is enough. I also keep it to at most one long model name per sentence.

The checklist: six steps for every script

1. Write the script with no English word at the start of a sentence and at most one long model name per sentence.
2. Apply the lexicon, longest keys first, with matched spans locked.
3. Let g2pW tag the rest, only where it disagrees with pypinyin, skipping 一/不, neutral tones and 括/和.
4. Use full-width punctuation.
5. Listen. For anything still wrong, render pinyin tag, homophone and raw text as a shuffled A/B/C test.
6. Write the winner back to the lexicon so every pipeline gets it.

Deep dive: voice drift, an error the check missed, speech-rate checks, and what didn't work

You can skip this section; the pronunciation setup above works without it. What follows is the investigation and measurement behind a few production changes, kept in detail for anyone, or any model, debugging the same pipeline.

A distant, off-pitch voice came from the Q4 route collapsing whole sentences, not from the machine move

On October 20 my narration sounded distant and off-pitch. The narration service had recently moved to another machine, so that was the first suspect.

I measured 10 videos, 30 lines, at three layers: raw TTS output, after post-processing, and the final video. One line stood out, line 2 of the CopilotKit short. This table compares it with the lines around it:

Measure	Outlier line	Neighboring lines
Raw loudness	-32.2 LUFS	about -17 LUFS
Spectral centroid	1717 Hz	about 500 Hz
F0 median (pitch)	159 Hz	about 120–129 Hz
CER from the whisper gate	0.083

The raw line was far quieter than its neighbors. Loudness normalization brought it up to the target level, and the noise floor came up with it. That is what "distant" sounded like. Its character error rate (CER) of 0.083 was inside the threshold, so the whisper gate passed it.

The machine move was ruled out on two points. The reference wav had an identical sha256 on both machines. And the mix stage was not the cause: the final video against the post-processed audio had cosine similarity of at least 0.9992, so mixing did not change the voice.

The real change was the inference route. Narration had moved from PyTorch, run with a prompt plus the reference audio, to a 4-bit GGUF model (Q4) in continuation mode, where the model picks up where the reference recording leaves off.

I ran a blind test on 6 clips. The 2 clips judged "weird" were both Q4. One was the CopilotKit line 2, distant. The other was GLM-Omni line 2, with the pitch off and the words unintelligible after 3 seconds. The other 2 Q4 clips and the 2 PyTorch clips sounded normal. So Q4 occasionally collapses a whole sentence, not every sentence. Both bad lines had already been published.

I switched back to Q8 and re-narrated the 14 published Chinese videos from the Q4 period, 109 lines in total, with a mean CER of 0.076.

I haven't isolated Q4 quantization from continuation mode. Both changed at once.

Current production runs Q8 fine-tuned on my voice. The exact file is whatever the service's health endpoint reports; I do not hard-code the version anywhere. The Q4_K_M file I linked in the previous post has since been replaced on Hugging Face by other quant recipes: a 4-bit version with protected layers, Q5_K_M and Q6_K. I haven't deployed any of them.

To catch the next collapse without a human, I added an automatic check. A line is an outlier if its high-frequency share is at least 0.08 and its spectral centroid is at least twice the median for the same video. An outlier gets re-rolled once automatically. If it is still an outlier, it gets flagged for listening, but it does not block the video. In calibration it had 0 false positives on 118 normal lines and missed one line where the voice had changed. That is a small sample.

The whisper gate passed a take that read the reference transcript aloud

On October 28, in a long video, one take's transcript contained the transcript of my reference recording: "這個詞聽不懂沒關係…我們開始" ("it's fine if you don't understand this word… let's begin"). The model had spoken part of its own voice prompt. The gate passed it.

I re-checked all 45 lines of the final cut with a separate whisper pass, plus a keyword scan for sentences from the reference text.

The gate should block two things: a transcript much longer than the script, and a transcript that contains any reference sentence. As of October 28 neither check was in the gate.

A speech-rate gate re-rolls lines instead of time-stretching them

Added November 4. Nothing gets time-stretched: no atempo, no rubberband. The visuals and subtitles follow the audio, so the audio itself has to come out at a steady pace.

Rate is measured in syllables per second:

• A Chinese character counts as 1 syllable, and so does a {pinyin} tag.
• English words count by syllables, acronyms by letters, and W counts as 3.
• Only the voiced range counts: 10 ms RMS frames at or above -45 dBFS, with silence at the head and tail trimmed.

Each line has to be within ±10% of the episode's median for the same language. If it is not, the same text is re-rolled with up to 4 seeds, and the take closest to the median wins. The baseline is locked when it is selected, for example 5.497382 syllables per second, so it does not drift as lines are replaced. If all 4 seeds fail, the pipeline stops for a human.

The same gate checks tag positions. At each {pinyin} spot, the character whisper heard there (after alignment) must be the original character or a homophone with the same tone. 幫我{zha2} ("fry it for me") heard as 抓 has to fail. 炸, 閘, 札, 紮 and 鍘 all pass, because they are all read zhá.

Lines without tags only get a gibberish screen: after stripping punctuation and normalizing Simplified and Traditional characters, the transcript has to be at least .48 similar to the script.

What didn't fix pronunciation: a voice LoRA and automatic tone checkers

LoRA on my own voice. I fine-tuned a LoRA on 4.5 minutes of my voice. In a three-way blind test on 10 lines there was no consistent winner. 知識 ("knowledge") was right in only one version. 步驟 was read with zhou in all three, where Taiwan says ㄗㄡˋ zòu. In a production A/B on 8 lines, the fine-tuned model won 3, the original won 1, and 4 were ties. Production switched to the fine-tuned model, but readings still rely on the lexicon.

Automatic tone checkers. I wanted a machine to tell me whether a tone was right. None was good enough:

Checker	Result
MFA (forced aligner) tones	2 of 4, effectively guessing
MFA initials and finals	4 of 4 on 垃圾, 訓練, 資安; failed on 步驟 zòu vs zhòu
whisper to toned pinyin, A/B choice, eSpeak	62.5%
whisper to toned pinyin, A/B choice, VoxCPM	72%
whisper, whole-word tone correct, free decoding	4–5%
tonara	50–63%

MFA could tell apart consonants and vowels (initials and finals) on 垃圾, 訓練 and 資安, but it could not tell zòu from zhòu, and that z/zh difference is exactly what I need it to catch. Its positive control was audio synthesized by eSpeak, not VoxCPM. None of these checkers is in production. That is why everything in this post fixes the text before the TTS, instead of trying to grade the audio after.

Also in this series: One RTX 2080 Ti Narrates Every Daily AI-News Short · Whisper CER as an Audio Quality Gate: 40 Lines, Three Traps