Synthetic Hospital Benchmark Finds Top AI Models Miss Half of Must-Know Patient Findings

Put 10 licensed physicians in front of ten patient charts, half of them real and half of them made up, and ask them to sort which is which. They get it right 53% of the time, about what a coin flip would manage. Give the fake charts to ten of today's strongest language models and ask them to rebuild each patient's problem list, and the best one tops out at a severity-weighted F1 of 0.73. The best human doctor in the comparison reached 0.89. When the models summarize a chart, they leave out roughly half of the findings a clinician is supposed to include.

Those are the main results from Synthetic Hospital, a benchmark from Christine Park, Valerie Chen and Tim Dettmers of Carnegie Mellon University, released this week as a preprint with code and data. It targets an obvious gap in medical AI testing. Models now do very well on licensing-exam questions, but that doesn't tell you whether they can read a patient's messy, months-long chart and figure out what matters. Park's accompanying blog post, "Finding the Corner Pieces," puts the problem in jigsaw terms: before anything else, a clinician has to find the few facts that hold the rest of the picture together.

A hospital built from exam questions

The benchmark has 1,268 synthetic patients with 5,602 encounters. It was built from public USMLE-style medical-education material, so there is no protected health information in it. Dettmers summed it up in his launch post on X: "1,268 patients, 5,602 encounters, zero PHI. Physicians could not reliably distinguish its charts from real ones."

The construction runs in the opposite direction from most synthetic-data work. The team starts with a knowledge graph. They pull diagnoses and findings out of board questions, map them deterministically to ICD-10-CM, SNOMED CT and LOINC codes, group related cases into longitudinal patients, and only then have a language model (Kimi 2.5) write the actual notes. Every label traces back to its source, and no two patients share source material. The authors put it this way: "it is the ground truth behind the record, not the record itself, that is complete." Real EHR data can't offer that, because a real chart only holds what someone happened to write down.

The records are served through an Epic-style simulator with FHIR R4 resources, OAuth2 and role-based access, and a 13-tool function-calling interface. There are four tasks: reconstructing the problem list, summarization (whole-patient, current-visit and specialty-specific versions), evidence retrieval, and inferring the clinical question behind an imaging order. Patients are divided into a 200-patient public split, a 268-patient held-out split, and an 800-patient training split with 7,619 instances. Every instance comes with a deterministic reward, so the training split doubles as a reinforcement-learning environment. The project is MIT-licensed on GitHub.

Where the models fall short

On patient diagnosis, Kimi 2.5-thinking led at 0.732. GPT 5.3 followed at 0.703, Gemini 3.1 scored 0.681, and Opus 4.6 scored 0.615. Gemma 3 27B trailed at 0.287. The top four were within 0.06 of each other. The authors read that tight cluster as a sign that "recent improvements in general-purpose frontier models have not yet produced measurable gains in diagnostic accuracy at the top end." No single model led every task. Opus 4.6 had the best whole-patient summarization score, and it was still only 0.550 finding-level F1. Evidence retrieval was the one clearly mature skill, with every model above 0.74 precision-at-5. Imaging-indication scores stayed near 0.5.

AlphaSignal's coverage of the paper notes that the main failure is omission rather than invention. Omission rates ran from 45% to 79% across models, and hallucination stayed near zero. In the authors' words, current frontier models "remain unreliable on these clinical tasks."

Putting the models in agent mode generally made things worse. In a test of GPT 5.3, Mistral Large and Llama 4 Scout on 100 patients, running inside an agent loop cut performance on summarization, retrieval and imaging, by as much as 0.24 on imaging for Llama 4 Scout. Failed sessions used an average of 145,000 input tokens, compared with 60,000 for successful ones. Patient diagnosis was the one exception: self-retrieving agents gained between 0.14 and 0.34 F1. The project README, however, warns that the paper's agent harness left the simulator's problem-list tool exposed, and that tool returns the diagnosis labels. As a result, "agentic patient-diagnosis scores obtained through that harness are inflated." The newer environment endpoints close that leak.

Why It Matters

Clinical AI has had a data problem it couldn't easily get around. Real records sit behind IRBs and data-use agreements, and even when researchers can get them, they have no complete answer key. Synthetic Hospital gets around both limits. Anyone can download it, and every label can be checked against a graph. That combination is what reinforcement-learning pipelines need. The results also point to a specific weakness. Models rarely make things up on these charts. Their problem is that they miss things, which in a clinical handoff can be just as dangerous. A benchmark that scores recall of must-include findings targets that failure directly.

There are caveats. The realism study covered only ten records and 100 judgments. The case mix comes from exam questions rather than real patient populations, so rare presentations may be underrepresented. The charts also don't simulate the contradictions and documentation errors found in real hospital records. The physician baseline comes from seven doctors on a 13-patient subset, and individual physicians ranged from 0.21 to 0.89 on diagnosis.

What to Watch

Watch the held-out leaderboard, and watch whether labs start training against the 7,619-instance RL split. That will show whether models can learn to find the corner pieces or whether the omission problem goes deeper than tuning can fix. Also watch whether agentic diagnosis gains hold up once they are re-run on the leak-free environment endpoints, and whether the team adds the messiness of real charts to the benchmark: missing notes, conflicting entries and coding errors.