Most hospital lead scores are never checked. A vendor or an agency builds one, the CRM shows a number from 0 to 100, counsellors call the high ones first, and everyone assumes it works. Six months later nobody can say whether a lead scored 80 is actually more likely to arrive for a consultation than a lead scored 40.
This article is for marketing leads and analysts at hospitals and clinic groups who already have a score, from a model or from hand-written rules, and want to know whether it earns its place. It covers four checks: whether the score ranks leads correctly (lift), whether its numbers mean what they say (calibration), whether it predicts the outcome that matters (arrival and revenue, not form fills), and how to retrain it from counsellor feedback. Every figure below is synthetic, generated to show the arithmetic, and is not a benchmark.
Prerequisites: a CRM export with a lead ID, the score at the time of first contact, the date of the enquiry and the eventual outcome. If your system overwrites the score as the lead progresses, fix that first, because you need the score as it stood when a counsellor decided whom to call.
Decide what the score is supposed to predict
A score without a target is decoration. Choose one outcome, define it precisely and fix the window.
- Arrival: the lead attended a first consultation within 30 days of enquiry.
- Conversion: the lead started treatment or a package within 90 days.
- Revenue: billed value within 90 days.
Arrival is the most useful starting point. It happens soon enough to measure, it is less affected by clinical decisions than treatment, and it is the point at which marketing spend has done its job. A score that predicts form completion, which is easy to measure, may predict nothing about who turns up.
Write the definition in the data dictionary: target_arrived_30d equals 1 if an attended visit is recorded within 30 days of enquiry_at, otherwise 0. Exclude leads whose window has not yet closed. Including recent leads makes everything look worse than it is.
Check 1: the lift chart
Lift tells you whether ranking by score puts the people who arrive nearer the top.
Method.
- Take all leads with a closed outcome window.
- Sort them by score, highest first.
- Split into ten equal groups (deciles) of leads.
- For each decile, calculate the arrival rate: arrivals divided by leads.
- Calculate lift as the decile's arrival rate divided by the overall arrival rate.
Synthetic example. Suppose 2,000 leads, of whom 500 arrived, an overall arrival rate of 0.25.
| Decile | Leads | Arrivals | Arrival rate | Lift |
|---|---|---|---|---|
| 1 (top scores) | 200 | 88 | 0.44 | 1.76 |
| 2 | 200 | 72 | 0.36 | 1.44 |
| 3 | 200 | 62 | 0.31 | 1.24 |
| 4 | 200 | 54 | 0.27 | 1.08 |
| 5 | 200 | 48 | 0.24 | 0.96 |
| 6 | 200 | 40 | 0.20 | 0.80 |
| 7 | 200 | 36 | 0.18 | 0.72 |
| 8 | 200 | 38 | 0.19 | 0.76 |
| 9 | 200 | 34 | 0.17 | 0.68 |
| 10 | 200 | 28 | 0.14 | 0.56 |
Read this table in three ways. First, the top decile has a lift of 1.76, so the score is doing something. Second, the arrival rates fall almost steadily, which is what you want. The slight rise from decile 7 to 8 is likely noise at this sample size. Third, even the bottom decile arrives at 0.14, so low-scoring leads still convert. This matters for policy: do not stop contacting them, just contact them differently.
Plot cumulative arrivals captured against cumulative leads contacted. If calling only the top three deciles in this example captures 222 of the 500 arrivals, you can say that a team with capacity for 30 per cent of leads would reach 44 per cent of those who would have turned up, compared with 30 per cent under random calling. Convert to counts when you present it, so nobody mistakes it for a promise.
Sample size. With under a few hundred closed leads, deciles will be jagged. Use quintiles, or pool several months. Compute a confidence interval for each band, or at minimum show the counts, so a reader can see how thin the data is.
Check 2: calibration
Lift cares only about order. Calibration cares about the numbers. If your CRM tells a counsellor "probability of arrival: 0.70", then roughly seven in ten such leads should arrive. A score that ranks well but is poorly calibrated misleads anyone who forecasts bookings, sets staffing or decides how much to spend per lead.
Method. Bin leads by predicted probability, for example 0 to 0.1, 0.1 to 0.2 and so on. For each bin, compare the mean predicted probability to the observed arrival rate. Plot predicted on the x-axis and observed on the y-axis. A well-calibrated model lies close to the diagonal.
The scikit-learn documentation on probability calibration explains reliability diagrams and shows how different model types distort probabilities: logistic regression is usually reasonably calibrated, while random forests tend to avoid extreme values and naive Bayes pushes toward 0 and 1. It also notes the Brier score mixes calibration with discrimination, so a lower Brier score does not by itself prove better calibration. Use the reliability diagram as your primary check.
Synthetic example.
| Predicted bin | Leads | Mean predicted | Observed arrival | Reading |
|---|---|---|---|---|
| 0.0 to 0.2 | 700 | 0.10 | 0.12 | Slightly under-confident |
| 0.2 to 0.4 | 600 | 0.29 | 0.24 | Slightly over-confident |
| 0.4 to 0.6 | 400 | 0.50 | 0.34 | Over-confident |
| 0.6 to 0.8 | 220 | 0.69 | 0.45 | Over-confident |
| 0.8 to 1.0 | 80 | 0.88 | 0.46 | Strongly over-confident |
(The bins add up to the same 2,000 leads and 500 arrivals as the lift table.)
This score ranks well but promises too much at the top. A forecast built on its face values would overstate arrivals. The remedy is calibration: fit a mapping from the raw score to observed rates on data the model was not trained on. The same page describes two options. Sigmoid (Platt) scaling is a simple monotone transform that works better with small samples, and isotonic regression is more flexible but needs more data and can alter the ranking by creating ties. For a hospital with a few thousand leads, start with the sigmoid, check the diagram again and move to isotonic only once you have more history.
If the score is a hand-built points system, treat the points as a ranking and calibrate by mapping each score band to its observed rate. That is all calibration is, and it makes forecasts honest.




