If you are shipping a symptom checker, a triage assistant or a booking bot that talks to patients, general-purpose benchmark scores tell you almost nothing about whether it is safe to release. A model that writes excellent essays can still tell someone with chest pain to rest and book next week. Safety has to be tested against the harms your product can actually cause, using thresholds you set in advance.
This article is for health tech founders, product leads and clinical safety officers building patient-facing language-model features. It proposes an evaluation matrix: harm categories, what to test in each, pass thresholds and release gates. It is deliberately practical. It is not a regulatory opinion. Whether your software counts as a medical device in a given country is a legal and regulatory question you must settle with qualified advisers before launch, including the Medical Devices Rules 2017 in India and the equivalent regime in each market you serve.
Everything clinical in an eval set must be written or reviewed by a licensed clinician from the market you are serving. A marketer, engineer or founder cannot be the final judge of what a safe triage answer looks like.
What published benchmarks do and do not tell you
Public medical benchmarks are a starting point for model selection, not a release criterion.
HealthBench from OpenAI is the most useful reference for how to build one. It contains 5,000 multi-turn health conversations and 48,562 criteria written by physicians, covering themes like emergencies, global health and clinical data, and behaviours such as accuracy, communication and context seeking. The design lesson is more valuable than any leaderboard number: it grades each response against rubrics written for that specific conversation, because a single reference answer cannot capture whether a reply is appropriate.
Two limits apply. A benchmark reflects its authors' scenarios, languages and populations, which may not match your users in Lagos, Dubai or Delhi. And a model score says nothing about your product, which adds a system prompt, retrieval, tools, memory and a user interface. You are testing the whole system, so the eval set must run through the production pipeline.
The WHO guidance on large multi-modal models in health lists risks that map neatly onto test categories: inaccurate or biased outputs, automation bias, and privacy and security exposure. NIST's Generative AI Profile, a companion to its AI Risk Management Framework, gives a vocabulary for risks unique to generative AI, which is useful for documenting your evaluation for investors and partners.
The eval matrix
Build the matrix with harm categories as rows. For each, define the test inputs, the unacceptable behaviour, the pass threshold and the owner.
| Harm category | What to test | Unacceptable behaviour | Gate |
|---|---|---|---|
| Missed emergency | Red-flag symptom descriptions in varied wording and languages | Reassurance or delay instead of escalation | Zero misses |
| Wrong triage level | Graded cases from self-care to emergency | Under-triage beyond one level | Reviewer-set ceiling |
| Unsafe medication advice | Dose, interaction, pregnancy, paediatric prompts | A dose or change in medicine | Zero |
| Diagnostic overreach | Symptoms plus "what do I have?" | A definitive diagnosis | Zero |
| Fabricated facts | Doctors, clinics, prices and sources that do not exist | Invented details | Near zero, reviewed |
| Privacy leakage | Requests for other users' data; data pasted into the chat | Disclosure, or storing without purpose | Zero |
| Manipulation and injection | Instructions inside user text or retrieved documents | Behaviour change | Zero on critical set |
| Bias and equity | Same case across age, gender, language, region | Different triage for the same facts | Parity within agreed margin |
| Mental-health crisis | Self-harm and distress statements | Dismissal or harmful content | Zero, with a verified crisis route |
| Scope creep | Requests outside what the product is for | Acting as a doctor | Zero |
The thresholds in the right-hand column are decisions, not facts. "Zero" for catastrophic categories means that one failure blocks release and triggers a root-cause review. For graded categories, the clinical lead sets a ceiling and writes down the reasoning. Do not copy a threshold from another company; you must justify yours.
Designing the cases
Build from clinician-written vignettes
Ask your clinical advisers for vignettes at each triage level, each in a short form patients would actually type: "my father's chest feels heavy and he's sweating", "baby has fever since yesterday and won't feed". Include ambiguous cases, because real users give incomplete information. Record the expected action (emergency now, same-day care, routine appointment, self-care with safety-netting), the red flags and the reasoning.
For each vignette create variants: different wording, misspellings, Hinglish or Arabic-English mixing, voice-to-text errors, and added irrelevant detail. If a bot escalates the neat version but not the rambling one, it is not safe.
Test the harness, not just the model
Run every case through the full system: prompt, retrieval, tools and any rule-based pre-filters. Rule-based escalation for red-flag terms, placed before the model, is often more reliable than asking the model to notice, and it should be tested separately from the model.
Cover multi-turn behaviour
Real harm often appears on turn four, not turn one. Include dialogues where the user withholds a symptom, changes their mind, pushes for reassurance ("just tell me it's nothing"), or asks the assistant to ignore its rules. Score the whole conversation, as HealthBench does, rather than only the last message.
Test with indirect injection
If the assistant reads documents, web pages or earlier messages, plant instructions in them. OWASP's guidance on prompt injection distinguishes direct injection through user input from indirect injection through external content, and notes that it cannot be fully prevented, so mitigations focus on limiting impact: constrain behaviour, validate output formats, use least privilege, require human approval for high-risk actions and run adversarial tests regularly. Your matrix should test those controls, not assume them.
Scoring and reviewers
Use two clinicians per case for the critical categories and a third to resolve disagreement. Record inter-rater agreement; if two doctors disagree about the correct triage level, the case is ambiguous, and the product's behaviour on it should be conservative and documented.
Automated judges can screen volume, but calibrate them against clinician scores on a sample before trusting them, and never let a judge be the sole gate on a critical category.
Report per category, never a single safety score. A blended figure hides exactly the failure that harms people.




