Skip to main content
NewNMC’s advertising guidelines for doctors are in force from 6 Oct 2026. See what to change, and get a free compliance auditNew NMC advertising rules for doctors: what to change
AI & Automation

Medical-Safety Evals for Patient-Facing LLMs: What to Test Before You Ship

A harm-by-harm eval matrix for startups shipping symptom or booking assistants: what to test, who scores it, pass thresholds and the release gates that block an unsafe launch.

Medical-Safety Evals for Patient-Facing LLMs: What to Test Before You Ship

If you are shipping a symptom checker, a triage assistant or a booking bot that talks to patients, general-purpose benchmark scores tell you almost nothing about whether it is safe to release. A model that writes excellent essays can still tell someone with chest pain to rest and book next week. Safety has to be tested against the harms your product can actually cause, using thresholds you set in advance.

This article is for health tech founders, product leads and clinical safety officers building patient-facing language-model features. It proposes an evaluation matrix: harm categories, what to test in each, pass thresholds and release gates. It is deliberately practical. It is not a regulatory opinion. Whether your software counts as a medical device in a given country is a legal and regulatory question you must settle with qualified advisers before launch, including the Medical Devices Rules 2017 in India and the equivalent regime in each market you serve.

Everything clinical in an eval set must be written or reviewed by a licensed clinician from the market you are serving. A marketer, engineer or founder cannot be the final judge of what a safe triage answer looks like.

What published benchmarks do and do not tell you

Public medical benchmarks are a starting point for model selection, not a release criterion.

HealthBench from OpenAI is the most useful reference for how to build one. It contains 5,000 multi-turn health conversations and 48,562 criteria written by physicians, covering themes like emergencies, global health and clinical data, and behaviours such as accuracy, communication and context seeking. The design lesson is more valuable than any leaderboard number: it grades each response against rubrics written for that specific conversation, because a single reference answer cannot capture whether a reply is appropriate.

Two limits apply. A benchmark reflects its authors' scenarios, languages and populations, which may not match your users in Lagos, Dubai or Delhi. And a model score says nothing about your product, which adds a system prompt, retrieval, tools, memory and a user interface. You are testing the whole system, so the eval set must run through the production pipeline.

The WHO guidance on large multi-modal models in health lists risks that map neatly onto test categories: inaccurate or biased outputs, automation bias, and privacy and security exposure. NIST's Generative AI Profile, a companion to its AI Risk Management Framework, gives a vocabulary for risks unique to generative AI, which is useful for documenting your evaluation for investors and partners.

The eval matrix

Build the matrix with harm categories as rows. For each, define the test inputs, the unacceptable behaviour, the pass threshold and the owner.

Harm categoryWhat to testUnacceptable behaviourGate
Missed emergencyRed-flag symptom descriptions in varied wording and languagesReassurance or delay instead of escalationZero misses
Wrong triage levelGraded cases from self-care to emergencyUnder-triage beyond one levelReviewer-set ceiling
Unsafe medication adviceDose, interaction, pregnancy, paediatric promptsA dose or change in medicineZero
Diagnostic overreachSymptoms plus "what do I have?"A definitive diagnosisZero
Fabricated factsDoctors, clinics, prices and sources that do not existInvented detailsNear zero, reviewed
Privacy leakageRequests for other users' data; data pasted into the chatDisclosure, or storing without purposeZero
Manipulation and injectionInstructions inside user text or retrieved documentsBehaviour changeZero on critical set
Bias and equitySame case across age, gender, language, regionDifferent triage for the same factsParity within agreed margin
Mental-health crisisSelf-harm and distress statementsDismissal or harmful contentZero, with a verified crisis route
Scope creepRequests outside what the product is forActing as a doctorZero

The thresholds in the right-hand column are decisions, not facts. "Zero" for catastrophic categories means that one failure blocks release and triggers a root-cause review. For graded categories, the clinical lead sets a ceiling and writes down the reasoning. Do not copy a threshold from another company; you must justify yours.

Designing the cases

Build from clinician-written vignettes

Ask your clinical advisers for vignettes at each triage level, each in a short form patients would actually type: "my father's chest feels heavy and he's sweating", "baby has fever since yesterday and won't feed". Include ambiguous cases, because real users give incomplete information. Record the expected action (emergency now, same-day care, routine appointment, self-care with safety-netting), the red flags and the reasoning.

For each vignette create variants: different wording, misspellings, Hinglish or Arabic-English mixing, voice-to-text errors, and added irrelevant detail. If a bot escalates the neat version but not the rambling one, it is not safe.

Test the harness, not just the model

Run every case through the full system: prompt, retrieval, tools and any rule-based pre-filters. Rule-based escalation for red-flag terms, placed before the model, is often more reliable than asking the model to notice, and it should be tested separately from the model.

Cover multi-turn behaviour

Real harm often appears on turn four, not turn one. Include dialogues where the user withholds a symptom, changes their mind, pushes for reassurance ("just tell me it's nothing"), or asks the assistant to ignore its rules. Score the whole conversation, as HealthBench does, rather than only the last message.

Test with indirect injection

If the assistant reads documents, web pages or earlier messages, plant instructions in them. OWASP's guidance on prompt injection distinguishes direct injection through user input from indirect injection through external content, and notes that it cannot be fully prevented, so mitigations focus on limiting impact: constrain behaviour, validate output formats, use least privilege, require human approval for high-risk actions and run adversarial tests regularly. Your matrix should test those controls, not assume them.

Scoring and reviewers

Use two clinicians per case for the critical categories and a third to resolve disagreement. Record inter-rater agreement; if two doctors disagree about the correct triage level, the case is ambiguous, and the product's behaviour on it should be conservative and documented.

Automated judges can screen volume, but calibrate them against clinician scores on a sample before trusting them, and never let a judge be the sole gate on a critical category.

Report per category, never a single safety score. A blended figure hides exactly the failure that harms people.

Want this applied to your practice? Get a free ai & automation audit.

Book my free audit

Release gates

Write the gates as a policy the team cannot quietly edit.

  1. Hard stops. Any failure in missed emergency, unsafe medication, diagnostic overreach, privacy leakage or crisis handling blocks release.
  2. Soft gates. Graded categories must meet the ceilings set by the clinical lead, and must not regress against the last release beyond a stated margin.
  3. Change triggers. Re-run the full matrix on any change to the model, system prompt, retrieval corpus, tools or escalation rules.
  4. Sign-off. A named clinical safety lead and a named product owner sign each release record, with the eval version, the results and open risks.
  5. Rollback. Keep the previous version deployable, and define who can switch it back and how fast.

After launch: monitor the real distribution

Pre-launch tests cover what you imagined. Production covers what users do. Sample conversations weekly for clinician review, with consent and data-protection controls in place. Track escalation rate, the rate of "I can't help with that" replies, user reports of harm, and drift when the model provider updates the underlying model. Add every real failure to the eval set as a permanent case.

Log what you need and no more. HIPAA does not apply to every market, and Google and Meta do not sign Business Associate Agreements, so if you are serving US patients take advice on which infrastructure and model vendors can be used with identifiable health data. For India, the DPDP Act and its Rules apply to personal data, and for the UAE and Saudi Arabia local health-data rules apply. Settle these with counsel.

What "safe enough" means for a startup

You will never prove zero risk. What you can show is that you identified harms, tested them with qualified clinicians, set thresholds before seeing results, blocked release when they failed and monitored afterwards. That record is what hospital partners, insurers and regulators will ask for, and it is also what your own team needs to resist shipping on Friday afternoon.

Keep claims modest. Do not describe an assistant as diagnosing, treating or "clinically validated" unless you have the evidence and the regulatory standing. If the assistant books appointments for international patients, the same discipline applies to the surrounding experience; our piece on what international patients look for before booking covers the trust signals on the site around it.

Where to start

  1. List the ten harm categories above and delete any that truly do not apply, with a written reason.
  2. Recruit two licensed clinicians for the markets you serve and agree their time per release.
  3. Write 20 vignettes per critical category, each with three variants.
  4. Decide hard stops and ceilings before running anything.
  5. Run through the production pipeline, score by category and store the record.
  6. Add monitoring and a rollback before launch.

For context, see our AI automation services, the AI chatbot service, the healthcare AI adoption statistics page and the HIPAA compliance guide. If your assistant answers on WhatsApp, AiTextBotz is one way to put the first response and a human handover on that channel; the eval matrix applies the same way.

Filed underllm medical safety evaluationmedical chatbot safety testinghealthcare llm evalssymptom checker llm testingpatient-facing AI release gates

About the author

Co-Founder & CTO · Gurugram, India

Arush builds the systems the rest of the company runs on — the patient-acquisition pipelines, the AI chatbots, the analytics that tie a single click to a booked appointment. If a campaign reports a number, it's because something his team built is tracking it.

Free guide · 15 pages

The Patient Acquisition Blueprint

A 90-day plan from first search to booked appointment.

The Pioneers Brief — one essay, three benchmarks, one teardown, every Thursday. Subscribe free →

Free ai & automation audit

Your next chapter starts with a conversation.

Three quick questions. We come prepared with a plan for your specialty and city.

  • A 30-minute call with a healthcare specialist
  • Your top three growth opportunities, in writing
  • Honest advice — even if that means not hiring us
  • No obligation and no hard sell
You’ll speak with a senior strategist
Consultations are hosted by Arush Thapar’s team.
Step 1 of 3Takes 30 seconds

What do you want to improve first?

Details stay privateNo obligationReply within 1 working day

Client portfolio · Healthcare websites

Work relevant to medical safety evals for patient facing llms what to test before.

Explore client work →