We Hosted MedGemma Three Times Before One Stuck

We've hosted MedGemma in three different places. The first was too expensive to leave running. The second wouldn't stay running. The third is the one we should have started with.

At Legacy, we are building the patient memory layer for healthcare, turning clinical conversations into longitudinal understanding so care teams can deliver more personalized, goal-aligned care at scale. One component of this is running patient conversation transcripts through MedGemma, Google's open medical model, to generate insights about the patient. MedGemma is an open-weight model, so we can choose where it runs or host it ourselves, but this is not as straightforward as it might seem.

MedGemma

We wanted to implement a medical LLM into our workflow because we suspected that it would pull out clinical context and constraints that a general-purpose model would miss. After doing some research on medical LLMs, MedGemma seemed to be the most powerful model that was open-weight, meaning that we could experiment with it without needing specialized access. We chose to use MedGemma 1.5 4B since it was the newest, highest rated MedGemma model, and at 4B parameters (vs. MedGemma 1’s 27B), it’s relatively inexpensive to serve. After making our choice, we just had to decide where to host it.

The first thing we learned was that "open model" and "easy to host" are not the same. We initially were looking for managed deployments, and the only option that supports MedGemma is Google Cloud’s Vertex. Without a managed host like Vertex, you would have to maintain and handle the weights and wire up the inference server, so using Vertex was a much easier starting point.

When we tested Vertex, it worked great.The problem is the billing model. A Vertex endpoint charges for as long as a model is deployed, whether or not a single request hits it. We found this out the hard way - an early always-on deployment ran up a few hundred dollars in only a few hours of runtime with almost no traffic. For a workload like ours, paying 24/7 for a GPU that works a few minutes an hour is backwards. Our traffic is bursty since it is tied to encounters between clinicians and patients. So we looked into serverless options, where GPUs could be spun up and spun down on demand, letting us pay per second of actual inference. That led us to RunPod.

Runpod

RunPod offers serverless GPU workers and a templated vLLM endpoint, which is what we wanted. Setup was quick, and the vLLM template got us a serving endpoint fast, but we hit several issues.

The pod-versus-serverless model wasn't obvious. Pods are persistent GPU instances you rent and manage yourself; serverless endpoints are autoscaling workers that spin up per request. We set up a serverless endpoint and then got prompted to configure a pod, which sent me down a "wait, which pod are we actually deploying now?" detour before the distinction clicked. 

There was an integration cost, too. RunPod's serverless API is job-based. You POST a job to a /run (async) or /runsync endpoint, with a request shaped to RunPod's own schema, then poll for status and dig the text out of a RunPod-specific response. Something like this:

// RunPod serverless input

{

  "input": {

    "prompt": "<|user|>\nWhat are the key signs of pneumothorax on a chest X-ray?\n<|assistant|>\n",

    "sampling_params": { "max_tokens": 512, "temperature": 0 }

  }

}

It works, but we had to redesign from the standard OpenAI API that most AI infrastructure providers use to fit the batch system, which was extra lift for no real gain.

Cold starts ran about two minutes. For a tool that needs an answer at the end of a live session, two minutes of the model waking up is not acceptable, so we built keep-warm pings to hold a worker open during sessions. That works, but it's a workaround for a latency problem, not a fix.

Then came the container errors. We started getting "error creating container" failures on the serverless vLLM endpoint tied to Docker storage options: XFS project quotas and a --storage-opt flag. The frustrating part was that the flag wasn't coming from our image. RunPod's own runtime was applying it underneath us, so we were debugging a failure we couldn't reach from my own configuration.

The endpoints also crashed repeatedly, and while Runpod support advised us that it was their fault, they didn't get us to a resolution in any reasonable time window. As a whole, RunPod was inexpensive but unreliable. We can work around a cold start. Unpredictable downtime we can't, because Legacy runs during live clinical conversations, and a host that drops out mid-session means the tool simply isn't there when a clinician needs it. Cheap GPU time stops being cheap the moment you're spending your own hours nursing it.

Modal

Modal is also serverless GPU, also supports vLLM, and bills per second. The difference is the simplicity. Modal provides a notebook-style interface to set everything up and you do not have to worry about configuring individual worker nodes like Runpod. Running MedGemma on Modal with vLLM gives you an OpenAI-compatible server at /v1/chat/completions. The OpenAI API shape has become the default way to talk to a language model, and most client libraries and SDKs already speak it, including the ones we were using elsewhere in Legacy. So even though MedGemma is Google's model running on a server we deploy, our code talks to it as if it were calling OpenAI. Pointing our existing client at Modal was a base-URL change, not a new integration. The same question from the RunPod section becomes: 

// Modal + vLLM, OpenAI-compatible

{

  "model": "google/medgemma-1.5-4b-it",

  "messages": [

    { "role": "user", "content": "What are the key signs of pneumothorax on a chest X-ray?" }

  ],

  "max_tokens": 512

}

The response comes back in the standard chat-completions envelope, so extracting the text is just choices[0].message.content. 

Our first pass served the model with plain transformers behind a custom FastAPI route, and that was the first mistake. Raw transformers give up continuous batching, paged attention, and KV-cache management, and it makes you invent your own API. vLLM does all of that and ships the OpenAI-compatible server for free. Lesson learned: serve with vLLM unless you have a specific reason not to.

The second mistake was the base image. We started from debian_slim with a pip-installed torch, which has no CUDA drivers, so you land on CPU-only inference or outright runtime errors. We decided to build on an nvidia/cuda base instead, so the GPU is actually available to torch and vLLM.

Then there's cold start, which is mostly the cost of loading model weights. The first working version kept the weights in a Modal Volume. That's network-attached storage, so every cold start streamed roughly 8GB over the network into GPU memory. The fix is to bake the weights into the container image at build time, with a download step that runs during modal deploy:

vllm_image = (

    modal.Image.from_registry("nvidia/cuda:12.x-devel-ubuntu22.04", add_python="3.11")

    .pip_install("vllm", "huggingface_hub")

    .run_function(download_model, secrets=[modal.Secret.from_name("access_medgemma_hf")])

)

After that, the weights sit on local storage when the container starts instead of arriving over the wire. The first deploy gets slower because it snapshots the model into the image, but every cold start afterward is much faster, which is the trade I wanted.

The last thing to understand is the idle knobs. scaledown_window controls how long a container stays alive after the last request. I run 15 minutes, so back-to-back requests in a session never pay a cold start, and the worker releases itself once the session is over. keep_warm=1 pins a container on permanently, which kills cold starts entirely but means paying for a GPU around the clock. For bursty traffic, the scaledown_window is the lever you want and keep_warm is the one that empties your wallet.

On cost, an L4 on Modal runs about $0.80/hr and bills per second. A request that uses eight seconds of GPU time costs around $0.002. With a handful of requests a day and a 15-minute scaledown window, that comes to roughly $5 to $15 a month. Pinning a container on 24/7 would be closer to $570 a month. The whole point of going serverless was to live at the bottom of that range, and per-second billing is what makes it possible. 

Migrating the Code

We switched from Runpod to Modal for the reliability Modal provided and, luckily,  Modal’s intuitive API spec made the switch bearable . On the RunPod side, we had built out a suite of custom utils for endpoint detection, config loading, input building, job call, polling, etc. Moving to Modal collapsed all of that into a single POST to /v1/chat/completions and a one-line read of choices[0].message.content. The keep-warm route, which on RunPod fired a real async job to hold a worker open, became a trivial chat completion with max_tokens: 1. Anywhere the app had hardcoded a RunPod endpoint ID, it now points at a Modal URL.

What We Learned

Managed hosting is nice and easy right up until you get the bill. Vertex is the only managed home for MedGemma, but you pay for it whether or not anyone's using it, which just doesn’t work for a lot of workflows.

Serverless GPU with per-second billing fits our needs, but that’s a newer landscape, and the provider has to actually stay up. The headline GPU rate is the cheap part. Reliability and support responsiveness are the part you pay for in your own hours, and that bill never shows up on an invoice.

When you do self-host, serve with vLLM, built on a CUDA image, and get the weights onto local disk so your cold starts aren't network transfers. And lean on the OpenAI-compatible API, because the day you want to move again (and that day always comes), you'll be changing a URL instead of rewriting your client.

MedGemma 4B has been serving Legacy's patient-insights path on Modal since the migration, at a fraction of what an always-on endpoint cost, and without us having to babysit a crashing worker. We took the long way around, but MedGemma finally has a home we don't have to think about. The best infrastructure is the kind you get to forget about.