Call QA on Every Call: A Scorecard for Hospital Call Centres That Replaces Sampling

This post is for hospital marketing and patient-access teams who own a call centre or an appointment line and want to know what happens on the calls their campaigns generate. Most centres review a thin sample of recordings by hand, often a few calls per agent per month. The rest is invisible. If you pay for call-generating ads, the invisible part includes the calls where a lead was lost.

Automated grading changes the arithmetic. A language model can read a transcript of every call and score it against a rubric in the time a human takes to open the recording. That does not make the model the judge. It makes it a first pass whose output humans check where it matters. This post sets out a scorecard you can adapt, the audit loop around it, and the mistakes that make automated QA misleading.

The scorecard here is a template, not a published standard, and nothing in it is a benchmark. Your medical and compliance teams should approve the compliance items before it is used on live calls.

Why sampling misleads

A small manual sample has two problems. It is statistically thin: a handful of calls per agent cannot reveal how often a particular failure happens, only that it happened. And it is biased by selection, because reviewers tend to choose long calls, complaint calls or calls from the same days.

For a marketing team, there is a third issue. The calls that matter most to campaign performance, such as first-time callers from an ad, are a minority of total volume. A random sample dilutes them. When the grader runs on every call, you can filter by source and read only the calls from the campaign you are investigating. Pair this with call tracking so each call carries a source label.

Principles before the rubric

Score observable behaviour, not impressions. "Agent was friendly" cannot be graded consistently by a person or a model. "Agent stated their name and the hospital name in the first ten seconds" can be.

Use binary or three-level items. Yes, no and not applicable are far more reliable than a five-point scale. Add a short evidence quote for every score so a reviewer can check it in seconds.

Separate what the agent controls from what they do not. A caller who hangs up before the greeting finishes should not reduce an agent's greeting score.

Keep compliance items distinct from soft skills. A missed consent statement is a different class of problem from a flat tone, and should be reported separately.

The scorecard

Five sections, each with a handful of items. Adapt wording to your language mix.

1. Greeting and identification

• Agent gave a greeting and the hospital or department name.
• Agent gave their own name.
• Agent used the caller's preferred language, or offered it.
• Agent stated, where your policy requires, that the call is recorded.

2. Intent capture

• Agent identified the caller's main reason for calling within the first few exchanges.
• Agent asked who the patient is, if the caller was not the patient.
• Agent captured speciality, doctor preference or symptom category in the caller's own terms, without giving a diagnosis.
• Agent confirmed their understanding back to the caller.

3. Booking and next step

• A concrete appointment, callback or next step was agreed.
• Date, time and location were read back.
• Agent captured a reachable phone number and confirmed it.
• Agent gave the fee or package information that the hospital has approved for sharing, or explained where to find it.
• If no booking occurred, the reason is recorded from a fixed list.

4. Compliance and safety statements

• Agent did not give medical advice, diagnosis or a treatment recommendation outside their role.
• Agent did not promise outcomes or guarantee a result.
• Urgent symptoms, if mentioned, were escalated using the hospital's protocol.
• Agent did not ask for or read out identifiers that the process does not require.
• Where your policy requires, agent recorded consent to be contacted by WhatsApp or call.

5. Close

• Agent summarised the next step.
• Agent asked if anything else was needed.
• Agent ended the call politely.

Do not turn this into a score of ninety items. About twenty to twenty-five is enough. Each additional item raises the chance of inconsistent grading and lowers the chance anyone reads the output.

How the grader should work

Treat the model as a structured extractor, not as a writer. For each call:

1. Input: the transcript with speaker labels and timestamps, plus call metadata such as source campaign, duration and language.
2. Output: for each rubric item, a value of yes, no or not applicable, an evidence quote and a timestamp.
3. Constraints: the model may only mark an item yes if it can quote the evidence, and must mark not applicable where the situation did not arise.
4. Separate outputs for call outcome (booked, callback, information only, lost) and a reason code.

Transcription quality limits everything downstream. A grader reading a poor Hindi or Tamil transcript will mark items wrong. Speak to your vendor about language coverage, and test the rubric separately by language. Where the transcript is weak, route the call to human review rather than accepting the score.

Voice platforms such as AiTelz handle conversations on the phone side, and the same thinking about rubric and audit applies to whichever tool produces your transcripts.

The audit loop: where humans earn their time

The point of automation is to reallocate humans from random listening to targeted listening. Build four review streams.

1. Calibration set. Have two reviewers score the same 40 or so calls independently, then compare to the model. Where reviewers disagree with each other, the rubric item is ambiguous and needs rewording. Where both disagree with the model, the prompt needs work.
2. Disagreement queue. Whenever the grader's confidence is low, or the evidence quote does not match the transcript, send the call to a person.
3. Risk queue. Every call that fails a compliance or safety item is reviewed by a human before anything is said to the agent concerned.
4. Random audit. A small random slice of passes is reviewed regularly to catch false passes, which are the silent errors.

Track the agreement between model and humans over time as a metric. When it drifts, usually after a script change or a new language, recalibrate.

Using scores fairly

An automated score is a coaching input, not a disciplinary verdict. Share the evidence quote with the agent. Review the rubric itself with agents, since they often know why an item punishes good behaviour. Do not rank agents on a score that is influenced by call mix: an agent who takes complaints will naturally score lower on booking items than one who takes new appointments. Compare like with like by intent and language.

Data protection and consent

Call recordings and transcripts are personal data, often with health information in them. The government's explainer on the DPDP Rules 2025 describes purpose limitation, data minimisation, storage limitation and security safeguards among the Act's principles. For call QA, apply them like this:

• Tell callers that calls are recorded and for what purpose, including quality review.
• Use the transcript for QA and coaching, not for unrelated purposes such as marketing lists, unless the caller has agreed.
• Decide how long recordings and transcripts are kept, and delete on schedule.
• Restrict who can open them, and log access.
• If a third-party model processes the transcript, check where it runs and what the vendor does with the data. Your legal team should approve this.

Where a provider is also recording calls for clinical records, other rules may apply that this article does not cover.

Pitfalls that make automated QA misleading

• Grader drift. A model update changes scores without any change in agent behaviour. Version your prompt and model, and re-run the calibration set after any change.
• Rewarding the script, not the call. If the rubric demands an exact phrase, agents will say it robotically and the model will reward it.
• Ignoring the lost-lead reason. The most valuable output for marketing is not a quality score but a reason code on non-booking calls: price, distance, doctor not available, wrong department, no answer. Count these by campaign.
• Treating one transcript language as typical. If your calls are in Hindi, English, Tamil and Hinglish, report scores by language.
• Alert fatigue. If every call triggers a flag, none will be read.

Where to start

• Pick one campaign line and grade every call from it for four weeks.
• Write the rubric at about twenty items, with yes, no and not applicable only.
• Calibrate with two human reviewers on a set of about forty calls.
• Set up the four review streams, with the compliance queue first.
• Report outcome reason codes to the marketing team weekly, by campaign.
• Review the rubric with agents after the first month.

Related reading: our page on AI call summarisation, the call tracking and analytics service and the problem page on front desk not converting calls.