Teaching an AI to Score Like a Clinician

Judging clinical conversations is a lot harder than it sounds. Even the strongest "off the shelf" AI models struggle with the nuance and empathy required to evaluate goals-of-care conversations well. These conversations hinge on moments most general-purpose models can’t reliably detect.

At Legacy, we are building the patient memory layer for healthcare, turning clinical conversations into longitudinal understanding so care teams can deliver more personalized, goal-aligned care at scale. To build a reliable product that could be used at MaineHealth and Dartmouth Medicine, we needed an AI rater that scores these conversations the way an experienced palliative care clinician would, with empathy where it's called for and discipline about what the rubric actually says.

In a baseline approach, the AI rater's scores had no statistical relationship to how trained clinicians scored the same conversations (r = 0.113). That gap matters because coaching feedback is only as good as the rater behind it. A rater that disagrees with clinicians gives learners feedback they shouldn't trust, and people figure that out fast, quickly losing faith in the AI coaching tool. A rater calibrated to clinician judgment gives feedback worth acting on.

After iterating on the prompt and the workflow around the model, we got to r = 0.854, p < 0.001. No fine-tuning. No added training data.The calibrated rater is now the engine of Legacy's coaching workflow.

The Goal

Our partners (MaineHealth and Dartmouth Medicine) needed an AI that scores conversations in a way that lines up with how a trained human rater would score the same conversation. This might seem like a simple ask, but when scoring qualitative moments, this can be more subjective and a deceptively hard target for an AI.

The 'trained human raters' are board-certified palliative care physicians and faculty clinicians, with a combined 35+ years of attending experience, from MaineHealth and Dartmouth Medicine who routinely conduct, teach, and supervise advance care planning (ACP) conversations. Together, Legacy and clinical leaders from both institutions developed a first-in-class evaluation rubric designed specifically to measure the quality, completeness, empathy, and patient-centeredness of ACP/goals-of-care conversations and define what constitutes a high-quality interaction. Prior to evaluation, raters align on this shared rubric and establish interrater reliability to ensure scoring consistency, creating a rigorous clinical gold standard against which the AI is evaluated.

Inter-rater reliability against trained clinicians is a two-sided constraint. If the AI consistently scores too generously, meaning that it hands out higher marks than a human rater would, clinicians don’t trust the praise. Every conversation looks great and the feedback is meaningless. If the AI consistently scores too harshly, marking conversations lower than a human rater would, clinicians stop using the tool because nobody wants to be told that they are consistently bad at their job by software that may not be scoring correctly. The product fails on either side of the target, so the only acceptable place to be is precisely where the clinician raters are.

From this, we partnered with four distinguished palliative care attendings to rework the evaluation rubric into a canonical rubric that gets fed to the model as a single source of truth. We took the full, 7-criteria rubric and encoded it as one artifact with atomic requirements that can be easily understood by AI models. We presented the model with a scale of how to judge, and this gave us our initial baseline. From there, we needed to find a way to get it to match our clinicians.

What we are measuring

Our calibration dataset was three recorded training videos of goals-of-care conversations, labeled R, S, and T. Each video showed a clinician walking a simulated patient through a serious-illness discussion. Both human raters and the AI scored these conversations against the same canonical rubric. There are seven criteria, each scored on a 0–3 scale, where higher scores indicate stronger demonstration of the criterion:

• R1 — Frames the Conversation. How the clinician opens the encounter and orients the patient to the discussion.

• R2 — Assesses Understanding. Whether the clinician asks what the patient already knows about their condition.

• R3 — Shares Prognosis. Whether and how the clinician communicates the clinical prognosis and its impact.

• R4 — Responds to Emotion. How well the clinician uses NURSE statements when emotion surfaces in the conversation.

• R5 — Explores Values. Whether the clinician elicits what matters to the patient about quality of life and goals of care.

• R6 — Summarizes Values & Priorities. Whether the clinician reflects the patient's values back to them in their own words.

• R7 — Makes Recommendations. Whether the clinician offers a care recommendation that explicitly references the values the patient shared.

NURSE, mentioned in R4, is a specific communication framework used in palliative care: Naming an emotion the patient is expressing, signaling Understanding, expressing Respect, offering Support, or Exploring further. Each letter is a distinct kind of emotional acknowledgment with concrete linguistic markers.

For reliability measurement we used the intraclass correlation coefficient (ICC), a standard agreement statistic in medical-education research. ICC measures how closely independent raters agree when scoring the same items: higher values mean more agreement. Values above 0.75 are typically considered excellent; 0.50 to 0.75 is good.

Testing

Before we measured the AI, we measured the humans. Three faculty raters scored all three videos across all 7 of our rubric items. The human ICC came back at 0.676 across all three videos and 0.744 on the two we focused on for our testing, videos T and S. That’s good overall, but it’s excellent for the test pair. In practice, this means three independent clinicians, scoring the same conversation against the same rubric, agreed closely enough that you could swap one rater for another without meaningfully changing the score. That same number was our north star for our model.

Then we ran the AI. The first pass came back at ICC = 0.696, which on paper looks almost identical to the human raters. However, when the statistician dug in, they found that the model had given 2s to every item in video T on all 3 independent runs. The ICC was being carried entirely by video S, where the model was at least responding to content.

Our correlation was much more illuminating at r = 0.113. Our model’s scores essentially had no relationship to the rater’s scores. It was drastically overscoring items R5 (Explores Values), R6 (Summarizes Values & Priorities), and R7 (Makes Recommendations). These are the three items that turn on whether a clinician explicitly links a recommendation, summary, or question back to something the patient said. We suspected that the model was scoring the topic of the conversation, rather than the actual behavior of the clinician. Or that maybe the model couldn’t work through the context of whether something actually linked back. R4 (Responds to Emotion) was a separate problem. It evaluates how well the clinician uses NURSE statements in their emotional responses, but it seems to score seemingly randomly. Our leading theory was that the model struggled to understand what a NURSE statement is.

Then we got to work on framing different solutions. We at Legacy know that input from our partner clinicians is crucial in the model iteration process. Their review of model output illuminated very precisely where things went wrong in the incorrect scores and allowed us to tweak the model, the prompts, and the context to address them. 

Differences across models

First, we experimented with different foundation models. We started with gpt-4o as a baseline, but we suspected that a higher thinking model might be able to better reason through the scoring logic. Switching to o3 with high reasoning showed an improvement to the rationales it produced, and gave us more accurate scores, particularly on rubric items R5 (Explores Values) and R6 (Summarizes Values & Priorities). Both of these ask the model to do something difficult: read an entire conversation, hold the patient's stated values in working memory, and then check whether the clinician circled back to those values later in the conversation. It’s more than pattern matching, but tracking a multi-step thread across a whole conversation, which a reasoning model like o3 can handle better than gpt-4o. Since o3 “thinks” before it answers, it was able to identify the patient’s values first, and then effectively gauge if they were referenced later and how. The extra reasoning steps gave the model room to do the cross-referencing that the rubric requires.

We experimented with several different models in our optimization before we landed on o3. Different foundation models behave differently, and so do different versions of the same model. This means to find the strongest for our use-case, we had to do significant testing.

The test setup held everything constant except the model. Same rubric, prompt, and videos. We performed 3 independent runs per model and we were looking to measure 2 things:

• ICC - does the model consistently score the same way across multiple runs?

• Total scoring delta- by how many aggregate points does the model differ from the target score? Conversations can have a total rubric score of 0-21, and the higher the delta, the worse the model performs compared to the clinicians.

The candidates were GPT-4o (our baseline), GPT 5.4, o3 with medium reasoning, and o3 with high reasoning.

GPT-4o had a high inter-run ICC but a large scoring delta. It was scoring consistently, but consistently wrong.

5.4 produced identical scores across all three runs (ICC = 1.000) and a meaningfully better scoring delta than 4o. Internally stable, but not accurate enough to clear the bar.

o3 with medium reasoning was the most interesting result. Its inter-run ICC dropped sharply to 0.669, lower than 4o, while its scoring delta landed in the middle of the pack. Medium reasoning gave the model just enough deliberation to break out of 4o's defaulting behavior, but not enough thinking budget to consistently land on the same conclusion across runs. The model was doing partial cross-referencing and was able to vary its answers, but not enough to reliably arrive at the same one twice. As a whole, it was directionally better than 4o, but too noisy to trust.

o3 with high reasoning resolved that. ICC of 0.951 and a scoring delta of 2.67, best in both dimensions. The extra reasoning budget gave the model room to identify the patient's stated values first, then scan the clinician's later turns to check whether those values get named back, and only then commit to a score. With enough budget, it converged on the same answer across runs. With less room to reason, the scores wandered.

We chose o3 with high reasoning for production calibration.

Tradeoffs

The choice isn't free. Reasoning models are slower and more expensive per inference than non-reasoning models. That's what you get for giving the model more tokens to think with. For Legacy's coaching workflow, the trade is acceptable. Scoring a goals-of-care conversation isn't a real-time interaction the way a chat response is. A clinician submits a recorded conversation, the system scores it, and the learner gets structured feedback minutes later. A few extra seconds of latency don't affect the user experience meaningfully, and the per-encounter cost is one a faculty-grade coaching workflow can absorb.

What we're actively monitoring is scale and vendor stability. Cost grows linearly with the number of clinicians and conversations, so as Legacy expands, the economics could shift. And while o3 (high) is the right pick today, model performance shifts with API updates from the vendor. 

Eventually, the model itself should be the easy part. As the rubric and the workflow tighten, the gap between high-reasoning and lower-cost models should narrow on our specific task. At that point, migrating is a config change, not a re-engineering project.

Filling the knowledge gaps

The poor performance on R4 came down to a knowledge gap. The model didn't actually know what a NURSE statement was, at least not as clearly as we needed it to.

The model had probably seen NURSE referenced in clinical literature, but the distinctions that matter in practice (what counts as Naming versus Understanding, what makes Respecting different from Supporting) live in faculty training, not in publicly indexed text. So the model defaulted to pattern-matching against the general shape of empathic language and hoped that was enough.

It wasn't. The model credited anything that sounded warm or attuned, whether or not it was actually one of the five NURSE techniques. "That sounds really hard" got marked as Naming, when no emotion word is being named. "I can see you're going through a lot" got marked as Respecting, when it's closer to a generic acknowledgment.

Fixing this took two ingredients. We added written definitions for each of the five techniques (Naming, Understanding, Respecting, Supporting, Exploring). Definitions alone let the model parrot back what Naming meant, but it still couldn't reliably distinguish Naming from Understanding on an actual transcript. So, we also added a canonical example for each, drawn from real clinical exchanges. The examples gave concrete linguistic patterns to anchor against. With both in place, R4 scores dramatically improved.

We expect this to recur with other specialized frameworks in the future, but we are more prepared to address it immediately now. Whenever a rubric uses a term from a specific clinical-communication tradition, the AI’s context needs the same treatment. We need to provide definitions for what the term means inside the rubric, examples of what it sounds like, and counterexamples for what it doesn't. This applies anywhere you're asking a model to apply a rubric built on specialized vocabulary, not just in clinical AI.

Uncertainty

The other major fix was addressing uncertainty. Several terms in the rubric were not quite doing the job that was intended. "Explicit" and "standalone" appeared throughout the scoring criteria without being defined. What counts as an "explicit" link between a recommendation and a value? What makes a value summary "standalone" rather than implied? Humans interpret these words differently, and the model was interpreting them generously. We added in-context definitions for what each meant inside the rubric.

We also told the model how to handle its own uncertainty. When it couldn't decide between two adjacent scores, the model was given the instruction to assign the lower one and to avoid crediting vague approximations. This was a deliberate counterweight to the original failure. The reason our baseline was so low ( r = 0.113) was that the AI treated ambiguous moments as "close enough," counting topic mentions as values exploration and warm-sounding lines as NURSE acknowledgments. Rounding down forced the model to score what the clinician actually demonstrated.

A fresh evaluation after these changes returned r = 0.854 (p < 0.001).

Closing

The win here wasn’t necessarily a smarter AI, but a tighter contract between our model and our human raters. Our measure of success is whether the rater inside our product can scale the nuance of trained palliative care faculty to every clinician who uses Legacy. At r = 0.854, it can. 

That's what makes the rest of the coaching workflow possible. Everything Legacy tells a user (which rubric item to focus on, which specific moment in a conversation to revisit, how their scores are moving across encounters) depends on the rater behind it being trustworthy. The calibrated rater is what runs Legacy's coaching workflow now. The work in this piece is what got us there. 

We’re not done, though. At times, the model still scores incorrectly on rubric item R6. We will go through the same iteration loop to fix it.

Eventually, when the AI system is stable enough that improvements to the harness stop moving the metrics we care about, we’ll consider reaching for fine-tuning. The motivation then will be cost and latency, not capability. The capability came from prompting and model choice, and that’s what kept rubric iteration rapid.

Reliability isn't an inherent property of AI. It's earned — through the rubric, the raters, the iteration loop. Clinicians don't trust a score because the model is impressive. They trust it because the system behind it was built to deserve that trust. That’s the kind of system we are building at Legacy.