How we pick a model for each job at Legacy 

At Legacy, we are building the patient memory layer for healthcare, turning clinical conversations into longitudinal understanding so care teams can deliver more personalized, goal-aligned care at scale. For each clinical conversation, we transcribe the audio, score it against different rubrics, and generate several different outputs to help the clinician better understand and care for their patients. Each of these steps and outputs is a separate job utilizing an AI model, sometimes multiple, and each of these jobs has its own stakes. Finding the right model and tool that’s scoped to the task can be difficult.

The easy move would be selecting whatever frontier model that OpenAI or Anthropic released most recently and route everything through it, but we don’t. We optimize every single step to balance output quality and latency. We treat model selection as a per-job decision, and we work to keep that decision easy to change as models change and easy to verify as the right choice.

Match the model to the job 

Each of the jobs in our clinical pipeline has a different baseline for what is acceptable. As an example, rubric scoring and coaching synthesis are two of many flows in our pipeline, and they require drastically different approaches.

Rubric scoring sits at the detailed end. It needs a model that can reason carefully about a transcript and hand back structured output that lands inside valid score ranges every single time; because an incorrect score is how a clinician stops trusting the whole product. So we use a slower, high reasoning model that can compare the conversation to the rubric and conduct an informed analysis. 

Coaching synthesis wants something else. It takes our scoring and turns it into concise feedback a clinician will actually read: clear, warm suggestions that sometimes include praise. So it runs on a lighter model with a little temperature to keep the writing from going flat. 

Other examples include our transcription and our patient narratives.We tested against half a dozen transcription providers and landed on AssemblyAI because of their strong diarization and medical context handling. A patient narrative is a few paragraphs that summarize how the patient is doing, how they feel, what they need, and other details of what’s important to them. Narrative generation is high volume and is built from already-synthesized insights, so it goes to a small, fast model like GPT-4.1 nano. Anything touching raw medical context we route to MedGemma on Modal, because a model trained on medical text behaves better there than a strong generalist does.

No single model wins all of these at once, if you pretend one does then you overpay on the cheap high-volume jobs while under-reasoning on the one a clinician's trust actually hinges on.

Keep the choice swappable

Every model assignment in our system reads from an environment variable, so they can all change at deploy time without touching code or cutting a release.

New models ship constantly, and we'd rather try one in staging the afternoon it drops than wait til the next sprint. We also run inside more than one health system, and what one wants tuned isn't automatically what the next partner wants. Per-deployment config lets us meet each of them without forking the codebase.

To make all of this easier, our central LLM wrapper knows a reasoning model from a standard one. When it sees a reasoning model, it hands over a larger token budget and sets an explicit reasoning effort, so we can swap one into a slot that used to hold a standard model without the call breaking on truncated output. 

Design for the model failing

Every model call can fail. We’ve seen outages, rate limits, context limits, and a litany of self-hosting problems. Now, nothing runs without a fallback.

Our self-hosted MedGemma model falls back to a general model, so a cold start doesn't stall an encounter. Transcription sits behind a provider abstraction, so we can switch to Deepgram or Groq if AssemblyAI goes out. Long audios that are too large for certain transcription providers automatically drop from the fast transcription path down to a sturdier one. We use the same pattern everywhere. We would always prefer the system degrade to something that still works rather than fail outright.

Know what the choice costs

Cost comes in two forms here, money and time, and both are easy to lose track of. Every model carries a per-token price, and a single encounter fans out into scoring, extraction, narrative, and sometimes live coaching, so the numbers climb quickly. We keep per-model pricing tables and a per-encounter cost tracker, so when someone evaluates moving a job onto a bigger or newer model, we can see what that does to our cost-per-encounter before it ships. 

Latency is an even higher priority. Post-encounter work (scoring, extraction, document generation) runs the moment the conversation ends and has to reach the clinician before they move on to the next patient. The right model for a job is the one that's fast enough to deliver before the moment passes.

Prove the choice before trusting it

We don’t just pick a model and call it good to go. Every choice goes through thorough testing and validation before we trust it.

When we’re evaluating a new model, we test it against a suite of several other models and compare on three axes: quality, speed, and cost. 

For quality, we compare outputs against our “golden encounters”, which are example encounters that our clinical partners have graded for us, so we have a true north star of what good looks like. For speed, we measure time for the model to return a usable result, per encounter. For cost, we roll each model’s per-token price up into a cost-per-encounter figure. Depending on the job, we may prioritize speed or quality, but we make our choices based on a combination of all of the factors. 

From there, we run everything through our automated testing gate. Our CI runs full patient encounters end-to end with Playwright, then checks what comes out. We confirm that the scores are in range, that the required fields are there, and, most importantly, that every claim is grounded in the real transcript rather than something the model invented. A model that hallucinates a value the patient never expressed fails the gate no matter how good its output reads. 

We’ve also built an evaluation harness for prompt and model changes. It re-runs our data extraction on a pre-determined set of golden encounters.The harness then uses a judge model to grade the output, and the recall and precision are recorded. The last step is a gate that decides whether to auto-promote a change, route it for human approval, or reject it. It does this based on how far it beats the current baseline. 

None of this replaces the clinicians. When one of them disagrees with a score, they can log negative feedback or overrides within Legacy, and those overrides become the backbone of the AI. Our rubrics and prompts adapt to the feedback, and the system continuously evolves to match what clinicians know is right.

The takeaway

Model choice is a decision we can make quickly, reverse cheaply, and verify constantly. Testing catches bad choices before they ship, fallbacks catch failures in the moment, and clinician feedback catches what both miss. We’ve built a system around fast, safe iteration, so that we can always keep the best product, with the best models for the job, in front of our users.