mlsanityData to 14 Oct 2026

LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

arXiv 2608.31016 ↗First posted 31 Aug 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the benchmark, prompts and judgements.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 8 of its 25 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

8 of 25 references.

References · 8

  1. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  2. 2025AbsenceBench: Language Models Can't Tell What's Missingcited by 9
  3. 2023Extrinsically-Focused Evaluation of Omissions in Medical Summarizationcited by 6
  4. 2026Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completenesscited by 4
  5. 2026Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responsescited by 2
  6. 2026Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluationcited by 2
  7. 2026CARE: A Conformal Safety Layer for Medical Summarizationcited by 2
  8. 2026Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Setscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.