Voice Agent Latency Budget for Hospital Call Centres: ASR, LLM and TTS in Hindi and Tamil

This post is for hospital operations heads, call-centre managers and healthtech product leads who are evaluating or already running an AI voice agent on an appointment line. It answers one narrow question: where does the time go between the moment a caller stops speaking and the moment the agent starts replying, and what changes when the caller speaks Hindi or Tamil instead of English?

Callers do not judge a voice agent on its accuracy dashboard. They judge it on the silence. A long pause makes people repeat themselves, talk over the agent or hang up, and each of those behaviours then breaks the next turn. Latency is therefore a design constraint, not a polish item, and it needs a budget that every component has to fit inside.

We do not publish latency benchmarks here, because a number measured on one provider and one network is not transferable. Instead the post gives you the budget structure, the stages to instrument, a test protocol for scripted calls in each language, and the decision rules for what to cut when the total is too high. The worked numbers are hypothetical and labelled as such.

What the caller experiences as one pause is five delays

A turn in a voice conversation looks like a single gap to the caller. Inside the system it is a chain:

1. Endpointing. The system decides the caller has finished. This is a silence threshold plus whatever model-based end-of-turn detection you use.
2. ASR finalisation. The speech-to-text stage turns the last audio into a final transcript. With streaming recognition, most words are already transcribed as interim results; the cost is the tail.
3. LLM time to first token. The model receives the transcript, the conversation state and the tool results, and starts generating.
4. Tool calls, if the reply needs a slot lookup, a doctor's availability check or a CRM query. These can dwarf every other stage.
5. TTS time to first audio. The speech synthesiser must produce the first playable chunk, not the whole sentence.
6. Network and telephony. Audio travels from the caller through the carrier or SIP trunk to your media server and back.

Add these together and you have the turn latency. Most teams track only the LLM and the TTS because those are the stages they can see in a vendor dashboard. Endpointing and telephony are usually the largest hidden terms.

A hypothetical budget to anchor the discussion

Suppose you set a target that the agent should begin speaking within about 1,000 milliseconds of the caller finishing, measured at the caller's ear. That is a design target, not an industry standard. One illustrative split, labelled as hypothetical:

Notice what this table leaves out: tool calls. A budget that has no line for a database lookup is a budget that will be broken on the first real booking. The practical answer is to treat tool-using turns as a separate class with a spoken filler and a higher allowance, and to keep the plain conversational turns inside the tight budget.

Where Hindi and Tamil spend time differently

Do not assume English numbers carry over. There are several mechanisms to measure, none of which we can quantify for your stack without testing.

Language and model coverage in ASR. Not every recogniser supports every language in every mode. Google's Speech-to-Text supported-languages page lists Hindi (hi-IN) and Tamil (ta-IN) by region and model, and the feature coverage differs by language and location. Indian-language specialists document their own coverage; Sarvam's documentation describes real-time and batch speech-to-text across 22 Indian languages plus English, including a code-mixed output mode. Check streaming support for your exact language before you design around it.

Code-mixed speech. A Hindi caller who says the department name in English is common. If the recogniser is locked to one language, that word can be mangled, and the transcript then triggers a clarification turn. A clarification turn costs a whole extra round of latency. A code-mixed or multilingual mode may cost a little more per turn but save entire turns.

Endpointing in Tamil. Tamil is agglutinative, so a single spoken word can carry what English spreads over several. Speakers also pause mid-clause. A fixed 250 ms silence threshold tuned on English can cut a Tamil caller off, which the caller experiences as the agent interrupting. Raising the threshold fixes the interruption and adds directly to latency. This trade-off is the main reason a single global endpointing value is wrong.

LLM token counts. Text in Indic scripts generally splits into more tokens per word than English text in the tokenisers of many general-purpose models. More input tokens raise time to first token slightly; more output tokens lengthen the stream that TTS must consume. This is a mechanism worth testing rather than a fixed rule, since tokenisers differ by model and change between versions.

TTS voice and language coverage. Not every voice is available in streaming mode in every language. Sarvam's docs describe text-to-speech voices for 10 Indian languages plus English, but whether a given voice streams audio chunks or returns a complete file is a property you must verify with your provider. A non-streaming voice puts the whole sentence's synthesis time into your budget.

How to run the test: 200 scripted calls per language

The only way to learn your real numbers is to measure them. The protocol below is the one we use as a method; it reports no results.

1. Write the script. Produce about 20 caller utterances per language covering the intents your line sees: book, reschedule, ask a fee, ask a doctor's availability, ask for directions, ask for a report. Include at least four utterances with a mid-sentence pause and four with English medical terms inside Hindi or Tamil speech.
2. Record native speakers. Use clips from several speakers, not one voice, with two or three background-noise conditions typical of a phone call. Clips must have the consent of the speakers.
3. Place calls through the real path. Dial the actual number over the actual carrier. Testing through a developer console hides the telephony term.
4. Log timestamps at each boundary. Record: last speech frame from the caller, endpoint decision, final transcript received, LLM request sent, first token received, first TTS audio chunk received, first audio frame played at the media server.
5. Repeat to 200 calls per language. With 20 utterances, that is about ten passes over the script. Report medians and the 90th and 95th percentile per stage, not averages. Callers remember the slow turns.
6. Compare within a language first, then across languages. A difference between English and Tamil only means something if the same intents and call conditions were used.

Run the whole protocol on AiTelz or on whichever platform you are evaluating, and keep the raw timestamps so you can recompute with different percentiles later.

Reading the results: decision rules

Once you have per-stage percentiles, apply simple rules rather than tuning everything at once.

• If endpointing dominates the 95th percentile, tune it per language and per intent. Slots like a phone number or a date of birth need a longer window than a yes or no. Many platforms let you set the silence threshold by turn type.
• If ASR finalisation dominates, check whether you are using the streaming API with interim results, or waiting for a full-utterance response. Google's streaming recognition guide describes streaming audio and receiving results as the audio is processed. Compare that against what your integration actually does.
• If LLM time to first token dominates, shorten the prompt and tool schema, route simple intents to a smaller model, and stream the output to TTS sentence by sentence rather than waiting for the whole reply.
• If TTS dominates, switch to a streaming voice, or begin synthesis on the first clause.
• If telephony dominates, look at where the media server runs. A server in a distant region adds round trips that no model choice can recover.
• If tool calls dominate, cache the data that rarely changes, such as department lists and standard fees, and call live systems only for availability.

Failure modes to watch for

Overlapping speech is the most common one. If the agent starts speaking and the caller continues, you need barge-in: audio from the caller must stop the synthesiser quickly. A hospital line also hears a lot of background noise, such as waiting rooms and traffic, and the voice-activity detector may mistake it for speech and cut the agent off.

Another failure is fast but wrong. Trimming the silence threshold to hit a latency target while the agent starts replying to half a sentence produces a worse call than a slightly slower one. Measure task completion, repeated-utterance rate and handoff rate alongside latency, and refuse any change that improves one at the expense of the others.

Finally, remember that callers on a medical line are often anxious or in pain. A slightly longer, calmer pause is more acceptable than a fast reply that talks over them.

Where to start

• Pick your single most frequent intent and measure only that, in both languages, before expanding.
• Add a timestamp at every stage boundary before changing any model.
• Set separate budgets for plain turns and tool-using turns.
• Tune endpointing per language and per slot type.
• Confirm streaming support for ASR and TTS in your exact language and voice.
• Review a sample of calls by ear with a native speaker, because a dashboard cannot hear an interruption.

For the broader picture of how voice fits into patient communication, see our guide on healthcare AI and automation, the AI voice assistant service and the page on intelligent call routing. To see how calls reach your marketing reporting at all, call tracking for healthcare covers attribution.