Building a Lead Scoring Model From 12 Months of Hospital Enquiries: Features and Leakage

Twelve months of hospital enquiries is a useful dataset: enough volume per department to learn from, and a full seasonal cycle. It is also a dataset full of traps. Most failed lead-scoring projects in healthcare do not fail because the algorithm was wrong. They fail because the label was vague, the features included information that did not exist at the time of the enquiry, or the validation split flattered the model.

This article is for analysts, CRM owners and digital heads who are building a scoring model for a hospital or a clinic group, or reviewing one built by a vendor. It assumes you know what a lead score is. If you want the introduction and the business case, read our pieces on prioritising patient enquiries with AI and AI lead scoring for healthcare practices. This one is about the modelling work: defining the outcome, engineering features and finding leakage.

Everything here is a method. The worked numbers are hypothetical and labelled as such. Patient data must be handled under your hospital's consent terms and the Digital Personal Data Protection Act; work on pseudonymised extracts and keep names, phone numbers and free-text clinical notes out of the modelling table.

Step 1: Decide what "good lead" means

The label is the most important decision, and hospitals have at least four candidate outcomes with very different meanings.

For most OPD-led funnels, "arrived within 30 days of enquiry" is the best starting label. It is observable within a month, it is not controlled by sales staff, and it correlates with revenue. For surgical or IPD funnels, "arrived" may be too early, and a second model for "treated within 90 days" is worth building once you have the volume.

Write the label definition down exactly: which event, within how many days, for which departments, and what happens to duplicate enquiries from the same person. Exclude records where the outcome is not yet knowable. A lead from last week cannot yet have arrived within 30 days, and treating it as a negative teaches the model that new leads are bad. This is a censoring problem, and the fix is to train only on leads old enough for the window to have closed.

Agree the label with the clinical operations lead. If the data cannot distinguish "booked" from "arrived" because the front desk does not mark attendance reliably, fix that capture before modelling. A score trained on unreliable arrival data will encode the data-entry habits of the staff.

Step 2: Build the modelling table at enquiry time

The rule that keeps a model honest is this: every feature must be computable at the moment the lead arrives, or at the moment you intend to score it. If you plan to score on arrival, that means the first minutes. If you plan to rescore after the first call, say so and build a second model with that information.

Create one row per lead, with a timestamp for the enquiry, the label, and features grouped as follows.

Source and channel. Channel (Google Ads, organic, Meta, WhatsApp, call, referral, walk-in), campaign category, landing page department, device type, and whether the enquiry came by form, call or message. Keep campaign names grouped to a stable set of categories; hundreds of campaign names create sparse categories that the model memorises.

Intent signals. The service or department asked about, whether a specific doctor was named, whether the message mentions a procedure or only a symptom, whether it includes a preferred date, and text length. For free text, use simple derived flags (mentions cost, mentions insurance, mentions urgency) rather than raw text, which may contain identifying details.

Contact quality. Valid phone format, whether the number matches an existing patient (a boolean, not the ID), whether the email is disposable, and whether the same number enquired before. Previous enquiry count is a legitimate feature if it counts only earlier leads.

Geography and access. City or pin-code band, distance band to the nearest facility, and whether the person is outside the hospital's catchment. For medical tourism funnels, country of origin.

Timing. Hour of day and day of week of the enquiry, and the season. Out-of-hours enquiries often behave differently from working-hours ones.

Response context at scoring time. If your system scores after the first response attempt, include whether the first call connected and the minutes taken to respond. These are legitimate only if the model is scored after that event.

Avoid features that merely encode how the staff behaved. Which agent handled the lead, for instance, can look predictive because strong agents are assigned better leads, and then the model rewards assignment, not quality.

Step 3: Find the leakage

Leakage is information in the features that the model would not have at prediction time. It gives impressive validation numbers and a disappointing live result. The scikit-learn documentation has a clear description of common leakage pitfalls. In hospital CRMs, the usual sources are these:

Run a simple test: train a model with one feature at a time. Any single feature that gives near-perfect prediction is almost certainly leaking. Then ask, for each remaining field, "at the moment of scoring, would a human have known this?"

Treat "target leakage through time" as a separate check. A person who enquires in January and again in March may have been treated after the second enquiry, and the CRM might back-fill the outcome onto both rows. Assign each outcome to a single lead or build the table from event timestamps.

Step 4: Split by time, then by person

A random split mixes late leads into the training set and early leads into the test set. The model then appears to predict the past from the future. Use a time-based split: train on months one to nine, validate on months ten and eleven, and keep month twelve untouched as a final test. For tuning, scikit-learn's TimeSeriesSplit gives rolling folds that respect order.

Within each fold, make sure the same person does not appear on both sides. Group by a hashed phone number or patient ID.

Do not assume 12 months is stationary. Campaign mix, pricing and seasonal demand shift. If performance in the last fold is clearly below earlier folds, the model is brittle, and you should look at which features drifted before trusting a deployment.

Step 5: Choose a model and evaluate for the decision

For tabular CRM data, start with logistic regression and a gradient-boosted tree model, and compare. A well-regularised logistic model is easy to explain to the call-centre head. Boosted trees often capture interactions such as channel by department. Use the simpler model unless the gain is clear on the held-out period.

Judge the model by what it changes. Accuracy is a poor metric when the positive class is small. Use these instead:

• Lift by decile. Sort held-out leads by score and compare the arrival rate in the top decile with the average. A working model shows a clear step down from top to bottom.
• Precision at the team's capacity. If callers can handle 200 leads a day, measure how many of the top 200 scores arrive.
• Calibration. If the score says 0.4, about four in ten of those leads should arrive. Calibration plots tell you whether you can use the score as a probability, for example to estimate expected bookings.
• Stability across departments and channels. A model that works for orthopaedics and fails for paediatrics should be split, not averaged.

A hypothetical example: suppose 1,000 held-out leads in a month have an overall arrival count of 150. If the top 200 scored leads contain 70 arrivals, the team reaches 70 of the 150 arrivals by calling 200 people first. That is an improvement in order of effort, not proof that the other 800 were worthless. Make the use of the score a prioritisation rule, not a gate.

Step 6: Use the score without creating harm

A score that decides who gets called first is a workflow tool. A score that decides who gets ignored can cause real harm, especially in a hospital, where a low-scoring enquiry may be a genuine clinical need.

• Never use the score to deny or delay a response to anyone describing urgent symptoms. Route urgent language to a human first, whatever the score.
• Contact every lead within your service-level window, and use the score to order the queue within it.
• Review sub-groups for unfair patterns, such as lower scores for certain languages or regions simply because historic follow-up was weaker there.
• Log the score and the reason codes, so that staff can see why a lead ranked where it did.

Step 7: Monitor after launch

Compare score deciles against actual outcomes every month. Watch for feature drift when a campaign changes. Retrain on a schedule, say quarterly, and always test the new model against the old on the same recent period. Keep a holdout, such as a small random share of leads called in normal order, so you can measure the real effect of prioritisation.

The scoring and outcome capture described here sit comfortably inside a CRM that records source, response and arrival cleanly; you can see how Pioneer CRM holds enquiries, ownership and follow-ups on the demo page.

Where to start

1. Write the label definition and the observation window.
2. Exclude leads whose window has not closed.
3. List features and mark each as known at scoring time or not.
4. Run the one-feature-at-a-time leakage test.
5. Split by time and group by person.
6. Evaluate lift by decile and precision at capacity.
7. Deploy as queue ordering with urgent-language routing, and monitor monthly.

For the platform side, see our AI lead scoring service, the glossary entry on lead scoring and the healthcare CRM guide.