Speaking Languages It Was Never Taught: Multilingual Voice AI, 2024 to 2026

Abstract

Two years ago, a synthetic voice that spoke convincingly in more than a handful of languages was a research result. Today a voice cloned from ten seconds of English audio can be deployed on a phone line in more than ninety languages, and as of September 2026 the leading commercial model deliberately gives that voice a native accent in each one. This paper reviews how the field got here. We trace the architectural line from neural codec language models through flow matching to large language model backbones, follow training data from tens of thousands of hours to more than five million, and set out the central design tension that remains (whether a voice speaking a new language should keep its own accent or sound native). We compare the language coverage and cross-lingual behavior of the major voice platforms, report a small in-house round-trip measurement across twenty languages on two ElevenLabs models, and close with the open problems that still separate "supported" from "solved," particularly for low-resource languages and for speech that changes language mid-sentence.

1. Introduction

A voice agent that answers a city's phone line in Spanish, Vietnamese, or Amharic depends on three systems working together. Speech recognition turns the caller's audio into text and decides what language it is. A language model decides what to say. Speech synthesis turns the reply back into audio in a voice the caller hears as one consistent person. This paper concerns the first and third of those, and mostly the third.

The question that organizes the paper is a practical one we face in production. When a voice that was recorded in one language is asked to speak another, what should it sound like, and how well does it actually do it? The answer has changed more in the last twenty-four months than in the decade before.

2. Three generations of neural speech synthesis

Single-speaker, single-language neural TTS. The neural systems that replaced concatenative synthesis in the late 2010s were trained per voice and per language. Quality was high, and scaling was linear in effort. A new language meant a new recording session with a native speaker.

Neural codec language models and zero-shot cloning (2023). VALL-E recast synthesis as language modeling over discrete audio codec tokens, trained on 60,000 hours of English, and cloned an unseen speaker from a three-second prompt [2]. VALL-E X extended the approach across languages, conditioning on source-language speech and target-language text, and reported that it "effectively alleviates the foreign accent problems, which can be controlled by a language ID" [3]. That sentence matters for everything that follows. It is the first clear statement that accent, in cross-lingual cloning, is a variable a system designer chooses. Meta's Voicebox showed that a non-autoregressive flow-matching model trained to fill in masked speech could do mono- and cross-lingual zero-shot synthesis with lower error and higher speaker similarity than VALL-E, and much faster [6].

Scale, factorization, and LLM backbones (2024 to 2026). The period this paper covers is defined by four developments.

The first is a split in architecture into a few families: codec language models (the VALL-E line, Fish-Speech [17], Llasa [19], Qwen3-TTS [28]); flow matching and diffusion (Voicebox, E2 TTS [12], F5-TTS [16], ZipVoice [23]); hybrids that generate semantic tokens with a language model and render acoustics with flow matching (CosyVoice 1 to 3 [13, 18, 22], Seed-TTS [9]); masked non-autoregressive models (MaskGCT [15], OmniVoice [30]); and next-token diffusion over continuous latents (VibeVoice [25]). The grouping is ours, built from the papers' own descriptions.

The second is explicit disentanglement of what a voice is from what it says. NaturalSpeech 3 factorizes speech into separate content, prosody, timbre, and acoustic-detail subspaces [8]. Spark-TTS splits semantic tokens from a fixed-length set of global speaker tokens [20]. MiniMax-Speech learns a speaker encoder from untranscribed reference audio [21]. Each is a route to the same goal, which is to let a voice's identity travel to a language its owner never spoke.

The third is replacing language-specific front ends with learned ones. E2 TTS works from raw characters with no pronunciation dictionary and no duration model [12]. Fish-Speech uses a language model for linguistic features rather than per-language grapheme-to-phoneme rules [17]. Removing the hand-built front end is what makes adding a language cheap, since the most expensive per-language component used to be the pronunciation layer.

The fourth is data, covered next.

3. One language or many

3.1 The data trajectory

Training sets have grown by roughly two orders of magnitude in three years. VALL-E used 60,000 hours (2023) [2]. MaskGCT and F5-TTS trained on about 100,000 hours of in-the-wild multilingual speech (2024) [15, 16]. Emilia-Large passed 216,000 hours of spontaneous speech across six languages [14]. CosyVoice 3 scaled from about 10,000 hours to one million, across nine languages and eighteen Chinese dialects [22]. Qwen3-TTS reports more than five million hours across ten languages [28].

The same period produced the opposite kind of scale. Meta's MMS covered 1,107 languages for synthesis in 2023, trained largely on readings of religious texts [4]. OmniVoice (April 2026) reports zero-shot synthesis in more than 600 languages from 581,000 hours of open data, and describes itself as having "the broadest language coverage to date" for zero-shot TTS [30].

Those two numbers describe the field's central imbalance. Five million hours spread over ten languages is roughly half a million hours per language. Five hundred eighty-one thousand hours spread over six hundred is under a thousand. The long tail is being reached, but it is being reached thin.

3.2 Why multilingual models help the long tail

A single model trained on many languages shares one acoustic and linguistic backbone. High-resource languages teach it how speech works in general, and low-resource languages borrow that knowledge. This is the transfer effect YourTTS demonstrated early, reporting usable results in a target language from a single-speaker dataset [1], and it is the logic behind MMS [4] and behind meta-learned language representations that aim at synthesis in languages with no training data at all [11].

Coverage grew accordingly. XTTS, in mid-2024, supported sixteen languages and noted in its abstract that most zero-shot systems at the time supported only one [10]. Two years later, commercial and research systems routinely list dozens to hundreds.

3.3 What monolingual voices still do better

The classic approach has not disappeared, and for good reason. A voice recorded by a native speaker carries that speaker's accent, rhythm, and regional features exactly. Until 2026, vendor guidance for older multilingual models was explicit. ElevenLabs' own help center says the model reads the language from the input text, while accent and pronunciation come from the voice, and recommends a voice cloned in the target language for the right accent [ElevenLabs help, same-voice-across-languages]. Its troubleshooting page notes that a non-native voice may "retain its native accent, or drift between different accents."

In other words, through v3, the text set the language and the voice set the accent.

4. Cross-lingual voice transfer and the accent question

4.1 The trade-off

Every cross-lingual cloning system has to choose a point on a curve. At one end, the voice keeps everything about its speaker, including the accent of their first language. At the other, the voice sounds like a native speaker of the new language, and some of the original identity is traded away to get there.

Recent work makes this precise. Accent Analogy Guidance (Cho and Lee, September 2026) observes that "the accent of the reference leaks into the target speech," shows that simply reweighting classifier-free guidance only moves a system along a fixed identity-accent trade-off, and proposes subtracting an estimated accent direction to land above that curve. It reports speaker-similarity gains on OmniVoice, MaskGCT, CosyVoice 2, and F5-TTS, with no gain on one system tested [32].

4.2 Two vendors, two answers

In late 2026 the commercial field has not agreed on where to sit.

ElevenLabs moved to the native end with Eleven v4 (September 28, 2026). Its documentation states that when the output language differs from the reference voice's language, v4 produces "fluent, natural-sounding speech in the target language" rather than carrying the reference accent over, using the example of a Korean-cloned voice speaking English as English "rather than English spoken with a Korean accent." In the same language, the original accent is preserved. ElevenLabs describes this as "a deliberate change in how cross-language generation works, not a bug," and says it is exploring a toggle [ElevenLabs v4 docs]. Its launch post attributes identity preservation to "a new method for capturing speakers' identities" and discloses no mechanism [ElevenLabs v4 blog]. We note that any claim that v4 uses a language-agnostic speaker embedding would be inference on our part; the vendor has not said so.

Hume took the opposite default with Octave 2 (October 2025): "When used to generate speech in a different language, Octave 2 predicts the speaker's accent" [Hume]. Resemble's open-source Chatterbox documents accent inheritance from the reference clip as a known caveat [Resemble]. Microsoft Azure handles accent per language through markup on its own stock multilingual voices, where a language tag "set[s] the preferred speaking accent of each language" [Azure SSML].

Which end is correct depends on the use. A brand voice that should sound like one recognizable person in every market may want the accent kept. A public service line answering a resident in their first language almost certainly wants native speech. For our purposes the second case dominates, which is why the v4 change matters to us.

5. The platform landscape

ElevenLabs is, in our assessment, the leading commercial platform for multilingual voice agents today, and the numbers support a careful version of that claim. It offers one of the broadest language ranges for a single cloned or designed voice in a production voice-agent stack, more than ninety languages as of v4, and it is the only vendor we found that documents native-accent cross-lingual speech as a deliberate design. It is not the broadest on every axis, and the table below is meant to show where others lead.

Platform	Synthesis languages (as documented)	Cross-lingual and code-switching behavior (as documented)
ElevenLabs v4 / v4 Turbo	90+ (91 codes listed)	Native accent in non-source languages, by design; mid-call switching in agents
Microsoft Azure AI Speech	154 locales across 82 base languages; 761 voices	Stock multilingual voices auto-detect 77 languages; accent set per language by markup
Google Gemini TTS / Cloud TTS	"Over 100" (Gemini 3.8 Flash-Lite); 87 locales in Cloud Gemini-TTS (24 generally available); 53 for Chirp 3 HD	Auto-detects input language; accent steerable by prompt
Amazon Polly / Nova 2 Sonic	41 language variants (Polly); 7 (Nova 2 Sonic)	Nova 2 Sonic documents mixing languages within a sentence
OpenAI realtime	Not stated; realtime translation outputs 13	Mid-sentence switching reported by press, not confirmed on a first-party page we could retrieve
Cartesia Sonic 3.6	44	Local pronunciation claimed; no cross-lingual clone claim
Hume Octave 2	11	Keeps source accent, by design
Deepgram Aura-2	7	Spanish voices switch with English
MiniMax-Speech	32	Speaker encoder from untranscribed audio
Alibaba Qwen3-TTS (open)	10	Highest speaker similarity in all ten languages against ElevenLabs Multilingual v2 (a previous-generation model)
Meta MMS / OmniVoice (research)	1,107 (MMS); 600+ (OmniVoice)	Breadth leaders; MMS voices are not zero-shot clones

Sources for each row are listed in the references. Units differ across vendors (languages, locales, voices) and we have kept each vendor's own unit.

Three points stand out. On raw breadth, open research systems from Meta lead by an order of magnitude, and Google's newest TTS claims more languages than ElevenLabs. On locale and stock-voice depth, Azure leads. On documented within-sentence code-switching, Amazon's Nova 2 Sonic and several recognition vendors make claims that ElevenLabs does not. Our own measurement (section 7) suggests ElevenLabs' models do handle mixed-language text in practice, even though the documentation does not claim it.

6. The listening side

Synthesis is half the loop. The agent can only answer in the right language if recognition heard the right language.

ElevenLabs Scribe v2 covers more than ninety languages for transcription, with an accuracy table that its own documentation divides into tiers by word error rate, from "Excellent" (5% or less) for thirty-six languages to "Moderate" (25 to 50%) for nineteen [Scribe docs]. Among commercial real-time recognition, that breadth compares with 32 languages for AssemblyAI's realtime model, 55 for Deepgram Nova-3 multilingual, and 55 plus bilingual packs for Speechmatics. Meta's Omnilingual ASR, a research release, covers more than 1,600 languages and reports character error rates below 10 on 1,237 of 1,570 evaluated languages [27].

Code-switching, meaning speech that changes language within a single utterance, is where recognition vendors are most actively competing. AssemblyAI documents "native, mid-sentence" switching on by default. Deepgram documents transcription of speech that "switches languages mid-conversation." Microsoft's VibeVoice-ASR research model reports native handling of switching "within and across utterances" [29]. ElevenLabs' explicit claim is narrower, covering English inside Indic-language speech on batch Scribe v2 [Scribe v2 upgrade].

For a voice agent, the practical consequence is that the language detection signal is only as good as the recognizer's error floor in that language. A recognizer with a 40% word error rate in Lao will, some of the time, deliver text to the language model that does not look like Lao at all.

7. In-house measurement

7.1 Method

We wrote one short front-desk script ("Thank you for calling. How can I help you today? Your appointment is on Tuesday at ten in the morning.") and translated it into twenty languages: thirteen widely spoken ones and seven we serve or expect to serve that have far less speech data behind them, three of which (Amharic, Lao, Wolof) were added to ElevenLabs' language list only with v4. We synthesized each with one voice on two models, Eleven v4 Turbo and Eleven v3, with no language hint, so each model had to infer the language from the text alone. We then transcribed every clip with ElevenLabs Scribe v2, also with no language hint, and recorded the transcript, the detected language, the detection confidence, and character error rate against the source (after removing punctuation, spacing, and case). All runs took place on November 9, 2026.

7.2 Results

Language	CER, v4 Turbo	CER, v3	Language detected from v4 Turbo audio (confidence)
English	0.00	0.00	English (0.96)
Japanese	0.00	0.00	Japanese (1.00)
Mandarin	0.00	0.00	Mandarin (1.00)
Arabic	0.03	0.03	Arabic (0.99)
Turkish	0.02	0.00	Turkish (0.99)
Portuguese	0.04	0.01	Portuguese (0.99)
German	0.05	0.05	German (1.00)
Vietnamese	0.05	0.00	Vietnamese (1.00)
Korean	0.06	0.06	Korean (1.00)
Italian	0.06	0.06	Italian (1.00)
Russian	0.09	0.00	Russian (0.95)
French	0.09 †	0.00	French (1.00)
Spanish	0.17 †	0.17 †	Spanish (1.00)
Hindi	0.25 †	0.25 †	Hindi (0.99)
Nepali	0.03	0.03	Nepali (0.73)
Somali	0.02	0.08	Finnish (0.06)
Swahili	0.00	0.00	Swahili (0.23)
Wolof (new in v4)	0.07	0.31	Italian (0.12)
Amharic (new in v4)	0.13	1.34 ‡	Amharic (1.00)
Lao (new in v4)	1.05 §	1.00 ‡	Thai (1.00)

† Formatting, not mishearing. The recognizer wrote "ten in the morning" as "10:00 a.m." (Spanish, French) and kept the English loanwords "call" and "appointment" in Latin script inside Hindi, which is the documented behavior of Scribe v2 for English inside Indic speech. The words were all recognized.

‡ Unintelligible. Amharic on v3 was transcribed in Devanagari script as nonsense, and Lao on v3 as Latin-script gibberish. Neither language is on v3's list, and these results show what an unsupported language sounds like.

§ Recognized, but in the wrong script. The v4 Turbo Lao audio came back as Lao words (including distinctly Lao vocabulary) written in Thai script, labeled Thai with full confidence.

7.3 Findings

The v4 additions are real. Amharic, Lao, and Wolof went from unintelligible or badly degraded on v3 to recognizable speech on v4 Turbo. For Amharic and Wolof the recognizer recovered nearly the full script; for Lao the content came through even though the recognizer chose the wrong writing system.

For widely spoken languages, the round trip is close to lossless. Once formatting differences are set aside, the thirteen widely spoken languages came back with few or no content errors on either model. v3 was slightly more exact than v4 Turbo on several European languages (French, Russian, Vietnamese, Turkish), which is consistent with v4 Turbo being the latency-optimized variant, though one script per language cannot establish that.

The language label fails before the words do. This is the finding that matters most for voice agents. For Somali, Wolof, and Lao, the recognizer produced a correct or nearly correct transcript and then named the wrong language (Finnish, Italian, and Thai). Swahili was transcribed perfectly with a confidence of only 0.23. In each case the failure lands on exactly the languages that have only recently become available. An agent that decides whether to switch languages based on the recognizer's language label would, for these callers, be deciding on a wrong answer, often a confident one. We return to this in Part 2 of this series.

Mixed-language text works, mostly. Separately, we asked each model to read one paragraph that moved through English, Spanish, French, German, Japanese, Swahili, and back to English in a single request. Neither vendor documentation nor any page we found claims this. Both models rendered all seven languages, and the recognizer transcribed six of the seven correctly from each. On v4 Turbo the Swahili sentence came back written in Arabic script. The same Swahili sentence on its own transcribed perfectly, so the failure depends on context and not on the language alone.

7.4 Limitations

This is a small measurement. Each language has one three-sentence script, read by one voice. The translations are ours and have not been reviewed by native speakers, which matters most for Lao, Wolof, Amharic, and Somali. A single recognizer serves as judge, so recognizer error and synthesis error cannot be fully separated, and the recognizer's own documented error rate in a language sets a floor under any result. Character error rate counts formatting differences as errors. We report it as an indication of where to look, not as a benchmark.

8. Open problems

Code-switching inside a sentence. Phrase-Localized Language-Contrastive Guidance (September 2026) starts from the observation that "current speech synthesis struggles with code-switching," with an inserted foreign phrase taking on the primary language's accent, and proposes a training-free per-phrase correction [31]. On the conversational side, CS3-Bench reports up to a 66% relative drop in knowledge question answering under Mandarin-English switching across seven mainstream speech models [26]. No vendor we reviewed publishes a quantitative intra-utterance code-switching benchmark for synthesis.

Accent leakage. Section 4 describes the trade-off. The measurement problem remains open, because accent classifiers with good accuracy exist for only a few languages [5], and recent work has turned to language-model judges and language-ID posteriors instead [32].

Prosody. SeamlessExpressive targets speech rate and pauses across languages [7]. Sesame's evaluation with conversational context found listeners still preferred human continuations, which suggests a prosody gap remains even in English.

Evaluation. Mean opinion score behaves as a relative score, sensitive to the set of systems in a test [33]. The de facto objective protocol measures intelligibility with Whisper or Paraformer and speaker similarity with a WavLM verification model [Seed-TTS-eval], and published numbers for the same model vary between papers with settings and normalization. TTSDS2 reports being the only one of sixteen metrics with correlation above 0.5 to human ratings in every domain tested, and maintains a refreshed multilingual test set [24]. Using a single recognizer as judge across ninety languages is biased toward the languages that recognizer handles best, a limitation that applies to our own measurement.

Data for the long tail. Before MMS, speech technology covered roughly one hundred of the world's more than seven thousand languages [4]. Community-sourced, compensated recording (as in Omnilingual ASR) and few-shot extension are the most promising routes we have seen. The gap between half a million hours per high-resource language and under a thousand hours per long-tail language is the clearest single measure of how far there is to go.

9. Conclusion

In two years, multilingual synthesis has moved from a research demonstration to a default. Any voice can now speak dozens of languages, and the leading commercial model has made a principled choice about how it should sound when it does. What has not caught up is evidence. Vendors publish language counts and preference wins, and almost no one publishes per-language intelligibility, accent nativeness, or code-switching results on a shared benchmark. For anyone deploying these systems to serve people in their first language, the honest position is that a language being supported means it is possible, and whether it works for a given community is something to measure, call by call.

About this series

This paper opens Language Week, five pieces on language and voice AI published November 9 to 13, 2026, moving from research toward practice. Part 2 covers how we prompt a voice agent to decide when, and whether, to switch languages on a live call. Earlier Workforce Wave research includes Every Call Counts, a 90,157-call reliability study, and The Opening Line, a 72,798-call study of business phone greetings.

References

Papers

1. Casanova, E. et al. YourTTS. arXiv:2112.02418 (2021).
2. Wang, C. et al. Neural Codec Language Models are Zero-Shot TTS Synthesizers (VALL-E). arXiv:2301.02111 (2023).
3. Zhang, Z. et al. Speak Foreign Languages with Your Own Voice (VALL-E X). arXiv:2303.03926 (2023).
4. Pratap, V. et al. Scaling Speech Technology to 1,000+ Languages (MMS). arXiv:2305.13516 (2023).
5. Zuluaga-Gomez, J. et al. CommonAccent. arXiv:2305.18283 (2023).
6. Le, M. et al. Voicebox. arXiv:2306.15687 (2023).
7. Seamless Communication et al. Seamless. arXiv:2312.05187 (2023).
8. Ju, Z. et al. NaturalSpeech 3. arXiv:2403.03100 (2024).
9. Anastassiou, P. et al. Seed-TTS. arXiv:2406.02430 (2024).
10. Casanova, E. et al. XTTS. arXiv:2406.04904 (2024).
11. Lux, F. et al. Meta Learning Text-to-Speech Synthesis in over 7000 Languages. arXiv:2406.06403 (2024).
12. Eskimez, S. E. et al. E2 TTS. arXiv:2406.18009 (2024).
13. Du, Z. et al. CosyVoice. arXiv:2407.05407 (2024).
14. He, H. et al. Emilia. arXiv:2407.05361 (2024); arXiv:2501.15907 (2025).
15. Wang, Y. et al. MaskGCT. arXiv:2409.00750 (2024).
16. Chen, Y. et al. F5-TTS. arXiv:2410.06885 (2024).
17. Liao, S. et al. Fish-Speech. arXiv:2411.01156 (2024).
18. Du, Z. et al. CosyVoice 2. arXiv:2412.10117 (2024).
19. Ye, Z. et al. Llasa. arXiv:2502.04128 (2025).
20. Wang, X. et al. Spark-TTS. arXiv:2503.01710 (2025).
21. Zhang, B. et al. MiniMax-Speech. arXiv:2505.07916 (2025).
22. Du, Z. et al. CosyVoice 3. arXiv:2505.17589 (2025).
23. Zhu, H. et al. ZipVoice. arXiv:2506.13053 (2025).
24. Minixhofer, C., Klejch, O., Bell, P. TTSDS2. arXiv:2506.19441 (2025).
25. Peng, Z. et al. VibeVoice Technical Report. arXiv:2508.19205 (2025).
26. Liu, H. et al. CS3-Bench. arXiv:2510.07881 (2025).
27. Omnilingual ASR team. Omnilingual ASR. arXiv:2511.09690 (2025).
28. Hu, H. et al. Qwen3-TTS Technical Report. arXiv:2601.15621 (2026).
29. Peng, Z. et al. VibeVoice-ASR Technical Report. arXiv:2601.18184 (2026).
30. Zhu, H. et al. OmniVoice. arXiv:2604.00688 (2026).
31. Lee, C. H. et al. Phrase-Localized Language-Contrastive Guidance. arXiv:2609.01016 (2026).
32. Cho, Y., Lee, J. Accent Analogy Guidance. arXiv:2609.29123 (2026).
33. Le Maguer, S., King, S., Harte, N. The limits of the Mean Opinion Score for speech synthesis evaluation. Computer Speech & Language. DOI 10.1016/j.csl.2023.101577.

Vendor documentation (retrieved November 9, 2026) ElevenLabs models; Eleven v4 docs and launch post (September 28, 2026); v4 Turbo in ElevenAgents; Scribe speech-to-text docs; Scribe v2 upgrade (April 2, 2026); help center pages on cross-language voices and accent drift; language detection tool. Microsoft Azure AI Speech language support and SSML. Google Cloud Chirp 3 HD and Gemini-TTS; Gemini 3.8 Flash-Lite TTS. Amazon Polly supported languages; Nova 2 Sonic language support. Deepgram TTS models and changelog. AssemblyAI multilingual streaming and code-switching. Speechmatics languages. Cartesia Sonic changelog. Hume Octave 2 launch (October 1, 2025). Resemble Chatterbox repository. Seed-TTS-eval repository. Full URLs in the source brief.