How do you write an end-to-end test for an app that listens?

Most end-to-end tests are very simple. Click the button. Assert that the expected text appears. Fill the form, submit, check the toast. 

The whole discipline rests on two assumptions: that you can drive the app with keystrokes and clicks, and that you know exactly what it should produce.

Neither of those assumptions held for us.

At Legacy, we are building the patient memory layer for healthcare, turning clinical conversations into longitudinal understanding so care teams can deliver more personalized, goal-aligned care at scale. In our workflows, a clinician starts a patient visit, the app records the conversation through the browser, and an AI pipeline turns that audio into live coaching during the visit, plus a series of different post-visit reports, scores, patient longitudinal insights, and other dynamic, often subjective and dynamic information.

All of our output relies on live audio. You can't type audio into a form field. And even a fairly deterministic language model's output is never identical between two runs, so expect(summary).toBe("...") is a non-starter. The two assumptions that make normal E2E testing easy were the two things we didn't have

Instead, we built a Playwright test that drives the real app in a real browser, feeds a fake microphone a recorded doctor and patient conversation, waits for the whole AI pipeline to finish, and then hands the output to a second AI to decide whether it's up to par. It is a complicated flow with some nuances that are tough to build around. Once we understood these nuances, we had a test that catches real regressions across the whole pipeline before they ship. Here is how each piece works, and what tripped us up along the way. 

The shape of the test

The core Playwright test runs a complete visit from login to the generation of the final reports. It takes roughly 10 minutes, with most of that being recording time. The sequence is fairly simple. Log in, create a patient, create an encounter, start the visit, start recording, let it run, end the visit, wait for processing, then open the reports and screenshot everything. What makes it worth writing about is that nearly every one of those steps hid a small landmine.

Take login. We started with Playwright's fill(), which sets an input's value in one shot, but it didn’t work for us. Our forms are built on React Hook Form, which listens for real keystroke events to fire validation and register the field. Setting the value directly skips all of that, so the form thought the fields were still empty. The fix was to make the test behave like a person at every step, simulating individual keystrokes and mouse movements.

await emailInput.pressSequentially(email, { delay: 50 });

pressSequentially types character by character with a delay, firing the same events a human keyboard would.

Getting sound into a headless browser

Sound was our biggest roadblock, and also the most important piece of testing the app. The app calls navigator.mediaDevices.getUserMedia() to grab the microphone for our live recording. In CI, there is no microphone to pull. 

After some digging, we learned that Chromium can fake audio input with a few (barely documented) flags. 

--use-fake-ui-for-media-stream        # auto-accept the mic permission prompt

--use-fake-device-for-media-stream    # swap real hardware for a fake device

--use-file-for-fake-audio-capture=/abs/path/to/test-conversation.wav

Now every getUserMedia() call in the app receives the contents of a WAV file as though someone were speaking into a mic. The app can't tell the difference, which is the point. We're not mocking the audio layer, we're feeding the real one. But there was a catch: Chromium's fake capture only takes WAV files at 16 kHz, mono, 16-bit PCM.Feed it anything else, and it just outputs silence without any errors or warnings.

After implementing the processing steps needed to get the fake mic working, we hit new issues around the audio’s timing. We had the fake mic working. The audio file was clean. And the recording captured the last two thirds of the conversation, every time. The beginning would just be completely cut out.

We learned that Chromium starts looping the audio file into the mic device the moment the browser launches, not when the mic is first requested. Our test needs thirty to sixty seconds to log in, create a patient, create an encounter, and actually start the visit. By the time recording began, the first minute of the conversation had already played and was completely missed by the app’s capture. 

After getting a handle on the test’s average time from browser launch to starting recording, we started padding all of our test audios with enough silence so that they wouldn’t start until afterwards. A small script uses FFmpeg to glue N seconds of generated silence onto the front of the real conversation and normalize the whole thing to 16 kHz mono. 

ffmpeg -f lavfi -t 45 -i anullsrc=r=16000:cl=mono \

       -i conversation.mp3 \

       -filter_complex "[0][1]concat=n=2:v=0:a=1" \

       -ar 16000 -ac 1 -sample_fmt s16 test-conversation.wav

This made it so that while the test is busy clicking through setup, the mic is emitting silence. By the time it clicks Start Recording, the real conversation is just starting. Then the test reads the WAV header to work out the file's true duration and records for exactly that long, so it stops the moment the conversation ends.

Manufacturing a conversation to test with

While we have some pre-recorded simulated patient/clinician conversations, sometimes we need to test a specific flow based on certain dialog paths. For example, we will auto-generate Advanced Directive forms for patients, such as living wills and medical power of attorneys, and we needed conversations where those were discussed. Our solution was to generate the conversations.

The pipeline has a few stages. A plain-text transcript goes to GPT-4o, which turns it into structured, speaker-labeled dialogue, clinician versus patient, with optional emotion tags like [gently] or [concerned]. That structured script goes to ElevenLabs, which synthesizes it with a different voice for each speaker. Having two voices matters for more than the realism of the conversation; it actually gives the downstream speaker diarization something real to separate. Then the resulting MP3 runs through the FFmpeg step above and becomes the silence-padded 16 kHz WAV the fake mic plays.

This means we can spin up new test conversations with specific characteristics, like a patient shifting toward comfort care or a mention of food insecurity, whenever we want to check that the coaching model captures and highlights them correctly.

Using AI to test AI

This test validates that the whole flow generates output, but the non-determistic nature of the system presents challenges in grading if it generates good output. Run the same visit twice, and you might get slightly different transcripts, two summaries worded differently, two sets of feedback that should be very similar but won’t be exactly the same. Exact-match assertions are useless. So how do we judge what’s correct?

We split it into two tiers, and split CI into two chained jobs to match. The first job runs the Playwright workflow and dumps everything (the database records, the coaching summary, the patient profile, and all the screenshots) into a JSON artifact. The second job only runs if the first passed, downloads that artifact, and validates it.

The first tier of validation is straightforward and deterministic. There are three strengths, one focused area for improvement, the overall score sits in range, the speaker metrics exist and add up. These catch the obvious failures.

The second tier is the interesting one. It sends the transcript and the AI's outputs to a smaller model, GPT-4o-mini at temperature 0.1, and asks it a long series of yes/no questions about meaning rather than strings. For example:

• Does the summary only mention topics that actually appear in the transcript, or did it invent things?

• Are the patient's stated values and goals grounded in what was actually said, or fabricated?

• Is the coaching feedback actually actionable, or just vague praise?

• Do the strengths and the areas for improvement contradict each other?

The model would output all of this feedback in a JSON object, and we would do a quick check if we were successful. This worked well and did exactly what we needed it to, so we still use this setup today.

What this actually bought us

We didn't set out to build something particularly interesting, just something that would tell us if our app was working and save us the extensive manual testing time. 

The typical unit test version would have told us the buttons work. This version tells us the audio still streams, the transcription still runs, the jobs still finish in the right order, the coaching model still grounds itself in what was actually said, and the report doesn't argue with itself. It catches the failures that live in between systems, something that’s difficult for any other automated test.