When to Switch: How We Prompt Voice Agents to Change Languages

The easy case is a request

When a caller asks for a language, the agent switches immediately. "Spanish, please." "¿Habla español?" "I don't speak English well." Any of those is a request, and a request is honored at once with no confirming question. Asking "Would you like to continue in Spanish?" of someone who just asked for Spanish wastes their time and sounds like the agent wasn't listening.

The detail that took us longest to get right is what counts as asking. A caller who names the language is asking. A caller who says they don't speak English is asking. The request can be spoken in any language, including the one they want.

The hard case is a caller who simply speaks

Most callers never ask. They just start talking. "Hola, necesito saber cuándo recogen la basura." That caller has told the agent what they need, in Spanish, and has asked for nothing.

Our first instinct, and the instinct of most prompts we've reviewed, is to switch. It feels responsive. In practice it goes wrong often enough to matter. Bilingual callers frequently open in one language and are perfectly comfortable in the other. Some callers greet in Spanish out of habit and then explain everything in English. And speech recognition sometimes hears Spanish where there wasn't any (more on that below).

So on this path the agent makes one offer and waits. The offer is spoken in both languages, so it lands whichever way the agent's guess went:

"Would you like to continue in Spanish? ¿Le gustaría continuar en español?"

Only a clear yes triggers the switch, and the agent records the caller's exact words of agreement when it does. If the caller says no, or answers in English, the agent carries on in English and does not ask again unless the caller produces more Spanish later.

One word is never enough

The single most useful rule in the whole set is a floor. A turn of one word is never treated as a language cue, however foreign it sounds and even when that word is the caller's entire turn.

"Hola" on its own tells you almost nothing. People say it to be friendly. People named Rosa say "Rosa." Background voices produce single words. Speech recognition produces stray single words from noise. Every time we let one word open the door, we got false offers, and a false offer in the first ten seconds of a call makes the agent sound confused.

With the floor in place, a one-word turn gets a plain answer in English asking what the caller needs. Two or more words of the caller's own business in another language is where the offer begins.

What the agent thinks it heard

Here is the part that most prompting guides skip, because it lives below the prompt.

A voice agent does not hear audio. It reads a transcript produced, in real time, by a speech recognition model that is also guessing the language. When that guess is wrong, the language model upstairs receives text in a language nobody spoke, and it has no way of knowing.

We see a handful of patterns again and again.

Background speech. A television, a radio, or a relative in the next room. Recognition transcribes it faithfully, and if it happens to be in Spanish, the transcript suddenly contains a Spanish sentence inside an English call.

Mumbling and crosstalk. Low, fast, or overlapping speech is the hardest audio there is, and recognition fills the gap with its best guess. Spanish is a common landing spot for garbled audio on our lines, likely because it is the second most common language in our call volume.

The agent's own echo. On some phone connections the caller's microphone picks up the agent's voice. If the agent just spoke Spanish, its own words can come back as a "caller" turn in Spanish.

Plain misidentification. In a test this week, we had a single synthetic voice read a paragraph that moved through seven languages, then ran the audio back through a transcription model. Six languages came back correctly. The Swahili sentence came back written in Arabic script. Same sounds, wrong language, delivered with full confidence. In a wider round of tests across twenty languages (reported in Part 1 of this series), the recognizer transcribed Somali almost perfectly and then labeled it Finnish. It did the same with Wolof, which it called Italian. The words were right and the language label was wrong, and the label is what an agent would act on.

The rule that handles all of this is short. Speech in another language that arrives with no caller behind it (a fragment inside an otherwise English turn, a broadcast, an echo of the agent's own words) is treated as unclear audio, never as a language cue. The agent answers the English it can understand, and if it understood nothing, it asks the caller to repeat.

The same rule covers a caller who turns away from the phone to speak to someone in the room. We staged that one as a mother who, mid-call, tells her son in Spanish to put his shoes on, then comes straight back to the agent in English.

By the letter of our rules, this should not have produced an offer. The agent treated an aside to a child as a language cue. It is also the clearest illustration of why the rule is to offer and never to switch. She said "No, English is fine," and the call carried on in English. A misread cost her one question rather than a call she could no longer follow.

Place names

This one surprised us. A caller says, in perfectly ordinary English, "I'm on Calle Mayor, near La Paloma Park, and the streetlight is out." To a language model reading a transcript, that sentence contains four Spanish words in a row.

Across the Southwest, Florida, and California, a large share of city names, street names, and landmarks are Spanish. Many of the cities we serve have them. An agent that treats Spanish words as a language signal will offer to switch to English speakers who are simply telling it where they live.

We handle this in two places. The rules say outright that a street, business, or personal name is never a language cue. And each agent's knowledge base carries the local place names it is likely to hear, so the model has seen "Calle Mayor" before the caller says it. A name the agent already knows is a name. A name it has never seen is a few foreign words, and that is when the false offer happens. We have found the second fix to matter as much as the first.

The caller who runs out of English

The opposite situation is the one the rules exist to catch, and it is the most common real switch we see. A caller starts in English, manages for a minute, and then hits something hard to explain. Midway through a sentence they give up and finish in Spanish.

Every exclusion above could suppress this if written carelessly. They said what they were calling about in English. They have been speaking English for a while. Neither should matter. Two or more words of the caller's own speech in another language is a cue to offer, whenever it arrives and however much English came before it. That rule sits deliberately after the exclusions so that it overrides them, and it triggers the offer, never the switch itself.

Staying there, and coming back

Once a caller has switched, every reply stays in their language until they move back themselves. Two things try to pull an agent back into English, and both need their own rule.

The first is tool results. When the agent looks something up, the answer usually comes back in English, because that is how the city wrote it. The agent translates the facts and answers in the caller's language. Without that line, agents routinely read the lookup result aloud in English and then apologize in Spanish.

The second is scripted language. A required notice ("this call may be recorded," a disclosure before a transfer) feels untouchable, so models tend to recite it word for word in English. Our rule is that "say exactly" fixes the content, not the language. The notice is delivered as a faithful translation.

Coming back to English works the same way going out does. A caller who asks to switch back gets English at once. A caller who simply starts bringing business in English is asked, in both languages, which they prefer. A one-word "okay" or "thank you" in English changes nothing.

Wiring it so the rules actually run

On the platform we build on, the switch itself is a tool call. The agent calls a language tool with the target language and a short reason. Two details there turned out to matter more than any wording in the prompt.

First, the tool's own description points at the exact heading of the language section in the prompt and says that section is the sole authority. When that pointer named a heading the prompt did not contain, the model fell back on its own judgment, and its own judgment is to switch on anything that sounds foreign. We now check that the heading matches before an agent ships.

Second, the reason field doubles as a diagnostic. The rules require a different quote for each path (the caller's request on the request path, the caller's agreement on the offer path). So when an agent misbehaves, the reason tells us which rule fired. A reason quoting the caller's opening greeting means the agent treated speaking as asking. A reason quoting a single word means the floor failed. That one field diagnosed a problem that four rounds of rereading the prompt did not.

Test it on a real phone line

Our worst version of these rules passed simulated testing and failed on its first live call. Simulation skips the speech recognition step, and the speech recognition step is where half of these failures begin.

So we test every multilingual agent on a real phone line, and we open with a question in the second language rather than a greeting. A greeting test is the easy one, and it is the test that let our earlier failures through. The question test makes the agent decide whether it has been asked for Spanish or simply spoken to in Spanish.

The rules are long, and longer than most people expect for what sounds like a simple behavior. Each one replaced a call that went wrong.

About this series

This is Part 2 of Language Week, five pieces on language and voice AI published November 9 to 13, 2026. Part 1, Speaking Languages It Was Never Taught, reviews two years of multilingual speech synthesis research and reports the twenty-language test cited above.