[Dev Workflow] Zero-Shot Voice Cloning on a MacBook: 24 Seconds In, Faster Than Realtime Out

There is a moment in every stock-photo search where you notice that none of these people work at your company. Preset TTS voices have the same problem, and it gets worse the more specific your voice is.

Narration sits on top of something you wrote, and the voice reading it is part of what people recognise. Narration sits on top of something you wrote, and the voice reading it is part of what people recognise. So I went looking, and I started with the defaults.

I did try the presets first. macOS ships say -v Meijia, which runs at 4.7 characters per second and is too slow to listen to. I auditioned preset voice libraries after that, and they all still sound unmistakably like a machine reading. There was no setting that would fix it, because what was missing was not a setting. So the route left was to clone my own voice, and the surprise is how little that takes.

Zero-shot cloning is not training, so there is nothing to prepare but one recording

No fine-tune. No hour-long recording. No weight update, no GPU rental, no waiting.

You hand the model two things: a recording of your voice, and that recording's exact transcript. It extracts timbre and delivery from the pair, then applies both to whatever new text you give it. That is the entire mechanism, and it runs in the same call that produces the audio.

One consequence falls straight out of this and shapes everything downstream. Because the model aligns the audio against the transcript, the two must match word for word. That is why a stumble means re-recording the whole take. You cannot trim the audio to fix a mistake without breaking the alignment, and you cannot fix the transcript to describe what you actually said without changing what the model thinks each sound corresponds to.

Install two packages, and let the model download itself on first run

python3 -m venv .venv && source .venv/bin/activate
pip install mlx-audio soundfile

That is the whole install. mlx-audio pulls in MLX, Apple's machine learning framework, which uses the Apple Silicon GPU directly. My versions: mlx 0.31.1, mlx-audio 0.5.1, Python 3.14 arm64.

The model weights are not in the package. The first load_model call pulls them from Hugging Face and caches them, so budget a download on your first run and nothing after that.

Record 20 to 30 seconds in one take, in the voice you actually talk in

Quiet room. Air conditioning not blowing at you. No background music. The MacBook's built-in mic is enough; I never used an external one. Sit about 15 to 20 cm from it.

Speak at your normal pace and volume, the way you would explain something to one person sitting across from you. Not a broadcast voice. Whatever you do here is what every sentence the model generates will sound like, so performing a voice you don't normally use means committing to it for the rest of the project. Twenty to thirty seconds is the target. Mine is 23.9 seconds.

Find the mic index first:

ffmpeg -f avfoundation -list_devices true -i ""

Mine reports [1] MacBook Pro的麥克風. Then record, trim the silence, and check the levels:

# the leading colon means audio only, no video device
ffmpeg -f avfoundation -i ":1" -t 30 -ac 1 -ar 24000 ref_raw.wav

ffmpeg -i ref_raw.wav -ac 1 -ar 24000 \
  -af "silenceremove=start_periods=1:start_threshold=-45dB:stop_periods=-1:stop_threshold=-45dB" \
  ref.wav

ffmpeg -i ref.wav -af volumedetect -f null /dev/null 2>&1 | grep volume

Mine comes back at mean_volume: -29.6 dB and max_volume: -10.7 dB. A peak between -6 and -12 dB is the range you want: loud enough that the quiet parts carry, far enough from zero that nothing clips.

Write your own reference text, and put the English terms in it

Write your own reference text rather than reading a standard passage. The point is coverage: it should contain the kinds of words you will actually be narrating.

Mine deliberately includes English technical terms in the middle of Chinese sentences — GPU, token, llama.cpp — because that is what my narration is full of. Read a clean literary passage into the mic and the model has never heard you switch languages mid-sentence, which is exactly the moment it will sound least like you.

Save that text to ref_text.txt, exactly as you said it. This file and ref.wav are a pair from here on.

The whole clone is ten lines of Python

from mlx_audio.tts.utils import load_model
import soundfile as sf

model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")
ref_text = open("ref_text.txt", encoding="utf-8").read().strip()

results = list(model.generate(
    text="<the new sentence you want spoken>",
    ref_audio="ref.wav",
    ref_text=ref_text,
    lang_code="auto",
    split_pattern="",
    verbose=False,
))
r = results[-1]
sf.write("out.wav", r.audio, r.sample_rate, subtype="PCM_16")

generate is a generator, so it needs list() around it, and the audio you want is the last item. split_pattern="" turns off the library's own text splitting, which matters because I do my own splitting one sentence at a time.

Two names in that snippet cost me time: 12Hz and Base

The model name contains 12Hz. The full name is Qwen3-TTS-12Hz-1.7B-Base, and those four characters are load-bearing. Leave them out and the call returns 401.

A 401 reads like an authentication problem. So you go check your Hugging Face token, then your login, then whether the repo is gated — and the actual fix is in the string you typed. If you get a 401 from this model, read the name character by character before you touch anything else.

The second name is Base, and it does cloning only. It has no preset voices at all. Preset voices live in the CustomVoice variant, and the two have different APIs, so code written against one will not run against the other. If you want presets rather than your own voice, you are downloading a different model, not passing a different argument.

83 seconds of narration takes 51 seconds, on the laptop

Faster than realtime, with no GPU machine involved anywhere in the path. That number is what makes the rest of the workflow practical: at this speed, regenerating a line is cheaper than deciding whether to regenerate it.

Know the pacing before you write a script, not after. My clone runs 5.3 to 6.5 characters per second, so 60 seconds of narration holds roughly 320 to 390 Chinese characters. If you are writing to a target length, that is the conversion factor.

Score it with whisper per segment, then listen to it anyway

The mechanical check is a transcription diff. Run each generated segment through whisper, compare against the script that produced it, and compute character error rate. Mine came out at 7.6%.

What that number catches is mispronounced words. What it does not catch is whether the result sounds good, which still needs your ears — the score can be clean on a take that is flat, or oddly paced, or subtly not you. I listened to mine. It passed.

Measure it per segment rather than over the whole concatenated file. That distinction is not a style preference; it changed my reading of this pipeline by a factor of 4.6, and the mechanism is in the deep dive below.

All three variants are Apache-2.0 and non-gated

Base, 1.7B-CustomVoice and 0.6B-CustomVoice are all Apache-2.0 and non-gated, so commercial use is fine.

Check it yourself before you build on top of it. Several comparable open TTS models are not commercially usable.

Deep dive: three variants, no seed, and a number that lied

Nothing below changes how you run the pipeline above, so skip it if you have what you came for. It is the measurement record: which variant does what, why the same sentence comes out differently every run, and how a full-file score lied to me by 4.6x.

Three variants, three different jobs, three different calls

I found the split between variants documented inconsistently, and what I measured differs from what several web sources claimed. Here is what the three actually do:

Variant	What it does	Call
Base	Cloning only. No preset voices.	model.generate(text=, ref_audio=, ref_text=)
CustomVoice	9 preset voices, plus an instruct emotion parameter	model.generate_custom_voice(text=, speaker=, language=, instruct=)
VoiceDesign	Describe a voice in words, get that voice	model.generate_voice_design(text=, language=, instruct=)

The nine presets in CustomVoice:

serena / vivian / uncle_fu / ryan / aiden / ono_anna / sohee / eric / dylan

The eleven languages:

auto / chinese / english / german / italian / portuguese / spanish / japanese / korean / french / russian

Base takes no instruct parameter because its delivery comes from your recording. Telling it to sound cheerful would be a second, competing source of prosody. If you want to direct the emotion, CustomVoice is the variant that accepts direction.

There is no seed, so the same sentence comes out different every run

What I saw. I regenerated a line I had already generated, expecting a byte-identical file, and got a different reading. The characters were right both times. The intonation drifted, and individual English letters came out differently from run to run.

What I assumed going in. That there was a seed parameter somewhere and I had not found it yet, since a seed is table stakes for a sampling-based generative model.

What I did. Went looking for one in the generate signature and the model config.

What happened. There is no seed exposed. Sampling is stochastic and this model exposes no seed to pin it down.

What the assumption missed. I had been treating run-to-run variation as a defect to eliminate. It is a property of the interface. Once that is settled, the question stops being "how do I make this deterministic" and becomes "how do I make re-rolling cheap", which has a much better answer.

A pronunciation dictionary would not have fixed anything

What I saw. Certain words came out mispronounced. Not every run, which I noted at the time and did not think hard enough about.

What I assumed going in. That the fix was a pronunciation dictionary: collect the words that came out wrong, pin each to the reading I wanted, apply it as a preprocessing pass.

What I did. Started sketching the dictionary, then went back to check how consistently each word failed, because a dictionary entry needs a stable wrong answer to correct.

What happened. The words did not fail consistently. The same word came out one way on one run and another way on the next.

What the assumption missed. A lookup table maps one input to one output. It has nothing to say about variance, and building it would have produced a component that could not do the job I built it for.

Re-rolling per sentence is what actually works

Split the narration into sentences and generate one file per sentence. When a sentence sounds wrong, re-run only that sentence.

This is only practical because generation runs faster than realtime. A single sentence takes a second or two, so re-rolling is effectively free, and you can keep pulling until a line lands right. It also decouples editing from synthesis: changing one line in the script means regenerating one file, not the whole take. On an 83-second narration that is the difference between a 51-second wait and a two-second one every time you change a word.

CER 0.35 on the whole file, 0.076 per segment

What I saw. My first character error rate reading on the 83-second concatenated file was 0.35. A third of the characters wrong is not a tuning problem.

What I assumed going in. That the approach was finished. I was ready to write off voice cloning as not accurate enough for narration and go back to presets.

What I did. Before dropping it, I looked at where in the transcript the errors were rather than how many there were.

What happened. They were all bunched at the end of the file. Whisper hallucinates at the tail of long audio, which is known behaviour. Transcribing per segment gave 0.076. A 4.6x difference, and it made usable audio look broken.

What the assumption missed. I had treated a bad number as a fact about the thing being measured. The broken component was the measurement procedure, and in the number those two are indistinguishable. That is the part worth carrying to the next measurement: an aggregate over a long file can hide a defect that lives in one region of it, and the aggregate is the form you are most likely to act on.

BreezyVoice looked like a shortcut and runs 24x slower than realtime here

BreezyVoice, from MediaTek, is a Chinese TTS model that also handles Chinese-English code switching. On paper it looked like a shortcut past the whole cloning step, so I tried it first.

On the MacBook it runs 24x slower than realtime. A minute of narration takes 24 minutes, which rules it out for a workflow where I regenerate individual lines while writing.

That is not a verdict on the model. It needs a GPU, and the premise of this pipeline is that it runs on the laptop I am already writing on. If you have a GPU box, measure it yourself rather than taking my number.

Wrapping up

A 24-second recording, two packages, ten lines of Python, and the narration comes out in your own voice, on your own laptop, under a licence that lets you sell what you make with it.

Three things not to skip: match the transcript to the recording word for word, keep 12Hz in the model name, and score per segment rather than per file.

What I am actually building with the narration is the next article.

---

Also in this series: The check in the skill was written correctly and never ran once · AI slop isn't the em dash, it's the structure