XTTSv2 anonymizer hits 0.49 EER

The paper shows XTTSv2 — a pretrained multilingual voice cloning model — can be turned into a speaker anonymization system with no retraining. The trick: swap the reference speaker embedding for a pseudo-speaker from a curated pool. What’s missing is any evidence the anonymization survives adversarial or fine-tuned ASV beyond the single ECAPA2 probe used here.

Design

Whisper-Large-V3 transcribes the input. XTTSv2 (Coqui TTS library) then runs five times, each time conditioning on a different pseudo-speaker from a pool of 10 per language and gender, constructed from Multilingual LibriSpeech. An ECAPA2 speaker embedding is computed on the original and each anonymized output. The iteration maximizing the harmonic mean of privacy (distance in embedding space) and utility (intelligibility proxy) is selected.

Measured	Baseline compared against	Sample	Conditions
EER (speaker verification)	SALT, MultiLingual	CV: 8,784 speakers; MLS: 272 speakers	7 European languages, dev+test splits
WER (intelligibility)	SALT, MultiLingual	Same	Same
ΔUTMOS (perceptual quality)	SALT, MultiLingual	Same	Same
CMOS (human preference)	SALT, MultiLingual	30 utterances, MLS English test set, 9 fluent English raters	270 ratings per system, 7-point scale

The ASV system behind every EER measurement is one ECAPA2 model. The paper notes that 50% EER equals random-guess performance — i.e., perfect anonymization.

Results

CommonVoice: average EER of 0.49, against 0.37 (SALT) and 0.46 (MultiLingual). MLS: 0.46, 0.30, 0.44. The authors frame the MLS figure as “comparably or better than MultiLingual,” but 0.46 versus 0.44 is a small loss. No confidence interval or minimum detectable effect is reported, so the gap’s statistical distinguishability from zero is not established.

WER on CV: 0.16 average, which the authors report as a 38% relative reduction over the baselines (SALT: 0.26, MultiLingual: 0.27). On MLS, SALT posts the lowest WER at 0.13. The system lands at 0.16 — ahead of MultiLingual (0.17), behind SALT by 3 percentage points.

ΔUTMOS on CV is +0.17 — the output rates slightly above the original — against −0.74 (SALT) and −0.62 (MultiLingual). MLS tells a different story: −0.35 for the system, still far less degradation than SALT (−1.29) or MultiLingual (−1.23). The human CMOS evaluation on English MLS gives −0.90, SALT −1.00, MultiLingual −1.85 on a −3 to +3 scale.

Reported distribution: iteration 0 chosen in 19.6% of cases, iterations 1–3 each roughly 18%, iteration 4 in 25.0%. The system doesn’t uniformly prefer the last pass. Early iterations frequently win the harmonic mean.

Reported by 2608.27360 exact values· scroll →

27k hours

XTTSv2 training data

0.49

average EER on CV

0.37

average EER on CV for SALT

0.46

average EER on CV for MultiLingual

Where the result stops

Seven European languages are covered (German, English, Spanish, French, Italian, Dutch, Portuguese). XTTSv2 supports 16 per the paper’s introduction. The other nine are untested.

Per-language MLS speaker counts are thin in places — 12 for Dutch, 20 each for Italian and Portuguese. Per-language EER estimates for those languages carry wide uncertainty the paper doesn’t quantify.

Privacy is measured against one ECAPA2 model. No fine-tuned ASV, no ensembles, no adversarial speaker verification. An EER of 0.49 against a single probe doesn’t license the claim that the output is unidentifiable against all reasonable attackers.

Human evaluation: English MLS only, 30 utterances, 9 raters. No inter-rater agreement or per-rater variance is reported. The CMOS figures are indicative, not precise.

No latency measurement. The refinement loop runs XTTSv2 up to five times per utterance; no wall-clock time is given. This is a batch pipeline.

What to do differently

For a system like nymic, which already runs a neural voice conversion pipeline on-device, the core architectural insight — that swapping the conditioning embedding changes perceived identity — is the operating principle. kNN-VC does exactly this: find a nearest-neighbor voice in a bank, condition synthesis on it. The paper validates that this class of approach reaches near-chance EER without task-specific training.

What it doesn’t validate: real-time operation, or resistance to a stronger ASV adversary. If the goal is a privacy guarantee rather than a perceptual identity shift, single-probe EER is insufficient. The actionable takeaway: repurposing works, degradation is manageable, iterative refinement is marginal when the pool is well-matched.

---

Method: this note was drafted by qwen/qwen3.8-27b from a single source — the arXiv abstract for 2608.27360. Before publication an automated gate re-checked every extracted claim against the source document (72 claim(s) and 41 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (0 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run 2608.27360’s benchmarks; figures attributed to it are its own.

FAQ

Does this mean a voice cloning model is inherently an anonymizer?

Not in the sense of a guarantee. Conditioning a voice cloning model on a different speaker embedding moves the ECAPA2 embedding close to chance-level separability. Whether that holds against a different or stronger ASV system is untested. The mechanism is the same one nymic already exploits for identity shifting; the difference is the evaluation target.

Can I use this to claim my voice conversion output is "anonymous"?

No. The 0.49 EER is against one specific ECAPA2 model on specific datasets. Anonymity requires either a formal privacy guarantee (e.g., differential privacy in the embedding space) or evaluation against a panel of ASV systems including fine-tuned ones. The paper provides neither.

Why does the system sometimes pick iteration 0?

The harmonic mean penalizes utility loss. If the first pseudo-speaker already produces a large embedding shift without degrading WER much, later iterations that shift further also degrade intelligibility enough that H drops. The 19.6% rate for iteration 0 suggests that for a meaningful fraction of inputs, the easiest anonymization is also the cheapest in quality.

Is the 38% WER reduction on CV a fair comparison?

The baselines differ in architecture. SALT uses WavLM self-supervised features; MultiLingual uses a GAN with Whisper-large-v3 and IMS Toucan. The system under evaluation uses XTTSv2 with Whisper-Large-V3 for transcription. The 38% figure is a relative reduction against those specific baselines on that specific dataset. It does not generalize to a claim about the WER floor of the anonymization task. If you want to explore the DSP side of voice synthesis conditioning, the Voice Synthesis DSP Sandbox is a reasonable place to experiment with embedding-space manipulation.