TL;DR: Answering the phone in another language is not a translation feature, it is a second version of your whole call flow. The speech model is rarely the hard part. The hard parts are deciding which language the call is in before the caller gives up, spelling names and addresses correctly, sending the follow-up text in the right script, and having an honest answer for what happens when the caller needs a human and nobody on shift speaks their language. Here is what changes when a voice agent goes multilingual, what breaks that never breaks in English, how to review a language nobody on your team speaks, and how to pick the languages worth supporting properly.
Most multilingual demos are the same demo. Someone calls, speaks Spanish, the agent answers in Spanish. Genuinely impressive the first time, and it tells you almost nothing about whether the system works on Tuesday afternoon with a real caller.
The gap is not linguistic. The voice engines are good and getting better on a schedule nobody controls. The gap is that a phone call is a workflow, and going multilingual duplicates every step of it: the greeting, the questions, the confirmation, the text afterwards, the CRM record, the person who calls back. Skip one of those and you have a bilingual greeting attached to an English-only business.
Translation is not the feature
The useful way to think about a second language is not "the agent speaks Spanish now." It is "there is a second path through the system, and it ends in the same place."
The booking has to land on the same calendar with a note about which language the caller speaks. The confirmation text has to go out in that language. The lead has to reach the CRM with a field telling whoever opens it what language to call back in. If the agent handles the conversation beautifully and then hands your team a record identical to every English one, you have moved the language problem downstream rather than solved it, and it surfaces at the worst moment, which is the callback.
There is also a content question nobody enjoys. Prices, service names, policies and hours have to exist in each language, written by somebody who knows both the business and the language. Machine translation of your service menu is a fine first draft and a dangerous final answer, because the words that get mistranslated are the specific ones: the name of a procedure, a warranty condition, what a deposit covers. It is the point the AI receptionist buyer's guide makes about every voice build. The content is the product and the model is the delivery mechanism.
The first five seconds decide the call
Language detection is the part people underestimate. The caller has to hear something they understand fast enough that they stay on the line, and you have three practical options.
A separate number per language. Crude, and it works. Dial the Spanish number, get a Spanish greeting, zero detection risk. The cost is marketing surface: two numbers to publish, track and register, and the caller who dials the wrong one.
A short bilingual greeting. The agent opens with a line in each of your top two languages, then continues in whichever the caller answers in. This handles the common case well and stays honest. It does not scale past two or three languages before the greeting becomes a menu that people talk over.
Detect and switch. The agent opens in the default language and switches as soon as the caller speaks. Best experience when it works, most demanding to build, because detection has to survive background noise, a two-word first utterance, and callers who open with "hello" out of habit before continuing in another language.
Then there is the case that catches every naive design: code-switching. Real bilingual callers mix languages inside one sentence, use English for the technical noun and their first language for the rest, or start in English and switch the moment the conversation gets financially or emotionally serious. An agent that locks a language at second three will lose exactly those callers. Mid-call switching has to be allowed, and has to survive being triggered twice.
One more pattern worth designing for: the proxy caller. An adult child calling for a parent, a neighbor helping, a coworker translating. The person on the phone is often not the person the appointment is for, and their preferred language may not be the language of the record. Ask, do not infer.
What breaks that never breaks in English
These are the failures that only show up once real callers arrive, and they are boringly practical.
Names and addresses. Spelling out a name is a culturally specific ritual. The spelling alphabet people reach for differs by country, letters that sound alike differ by language, and a name that is unambiguous in speech can have three plausible written forms. Street names are worse, because the caller may say the street the way the neighborhood says it while your map data holds something else. Readback confirmation matters more here than anywhere.
Numbers, dates and times. Phone numbers are grouped differently by country, dates are ordered differently, times are spoken differently. Each is a chance to capture a correct-looking value that is wrong.
The text message afterwards. If the confirmation goes out in English, the call did not finish. There is also a technical detail with a real cost: messages in non-Latin scripts are encoded differently and fit far fewer characters per segment than plain Latin text, roughly 70 instead of 160, so a one-segment English message becomes three. That is a billing line and a truncation risk, not a rounding error.
Pronunciation of your own proper nouns. Your business name, product names and street names are usually English, and they now sit inside a sentence in another language, where the synthesized voice will happily mangle them. This is fixable with pronunciation tuning, and it is the single most common reason a fluent-sounding agent sounds slightly wrong to a native speaker.
Register and dialect. Formal versus informal address changes how a business sounds and should be decided deliberately rather than inherited from whatever the model defaults to. The same applies to regional vocabulary: the word your callers use for an appointment, a quote or a deposit may not be the word a general-purpose model picks.
The handoff nobody plans
Here is the question that decides whether a multilingual deployment is real: what happens when a caller in language X needs a human, and no human who speaks X is on shift?
Transferring them to somebody who cannot help is the worst available outcome, because the agent has already established that this business speaks their language and the transfer proves otherwise. There are a few honest answers, and each is a business decision rather than a technical one.
Route to a bilingual member of staff during the hours they actually work, and to a different path outside them. Or capture the request completely and promise a callback in a stated window, with the language written into the record so the right person makes it. Or route to an interpreter service if you already use one. What matters is that the choice is deliberate and the agent never promises a transfer it cannot deliver. Our guide to the missed call revenue leak covers the general version, and language raises the stakes because the caller has fewer alternatives.
Reviewing a language nobody on your team speaks
This is where multilingual deployments quietly fail. Everything in the voice agent monitoring loop still applies, with three additions.
Somebody who speaks the language has to read the transcripts. Not the model, not a translation of them, a person. Translation hides the exact things you are looking for: the wrong register, the stiff phrasing, the term no customer would use. If you cannot arrange that review for a language, you are not ready to launch it.
Metrics have to be split by language. An aggregate resolution rate is the perfect hiding place for one broken language, especially a low-volume one, because the number stays healthy while every caller in that language hangs up. Look at resolution, escalation and abandon point per language and compare each against itself over time.
And the regression set has to be per language too. A change made to fix an English flow can alter behavior in another one, since the underlying prompt and logic are usually shared. Re-run both.
Which leads to the honest way to choose languages. Supporting 100 is a capability, not a deployment. Pick the languages your callers actually speak, from your own call logs rather than a list, and tier them: a full booking flow in the two or three that carry volume, a capture-and-callback flow in the long tail. In a language you cannot review, "I can take your details and have someone call you back in your language" beats a full booking flow that mishandles it. If your organization receives public funding, check what language access obligations come with it before you scope, because that changes which tier a language belongs in.
Buyer checklist
- How is the caller's language detected, and what happens if detection is wrong?
- Can the agent switch language mid-call, more than once?
- Do confirmations, reminders and texts go out in the caller's language?
- Is the caller's language written into the calendar entry and CRM record?
- What happens when a caller needs a human and nobody on shift speaks their language?
- Who reviews transcripts in each language, and how often?
- Are resolution and escalation rates reported per language or only in aggregate?
Where we fit
We build the AI voice and SMS systems behind Fortell AI, which helps Community Action Agencies answer every call and simplify intake in 100-plus languages, so their teams can spend time on people rather than paperwork. It is a demanding environment for multilingual voice: callers often have no easy alternative, and a wrong answer has real consequences for the person on the line. The same core runs behind CallGuard AI and CallSetter AI, answering, qualifying and booking around the clock.
Adding a second or third language to an existing agent, or building one where multilingual is a requirement rather than a nice-to-have, we handle as one project: the flows, the content, the integrations, the follow-up messaging and the review loop, on the ship-in-days approach we bring to everything. If the need runs past a standard receptionist, it is the same team you would hire for custom AI agent development.
Losing callers at hello over language? Book a demo and we will map the languages your calls actually come in, what a full flow versus a capture flow should cover for each, and what the handoff has to look like. See our work: Fortell AI, CallGuard AI, CallSetter AI and more, shipped in days, not months.