mlsanityData to 14 Oct 2026

One note in three: a verified census of three deployed AI scribes, and the instrument that counted it

arXiv 2608.31017 ↗First posted 31 Aug 2026cs.CL cs.AI cs.CYcited by 1

Not on the map. Why?

Abstract (arXiv)

Ambient AI scribes draft clinical notes under the reassurance that a clinician signs every note. We audited three commercial AI scribes on the same 142 consultations: 565 notes from recorded UK primary-care and US ambulatory encounters plus authored scenarios. Twelve discovery passes proposed 13,678 candidate errors; the 5,898 clearing an importance filter went to an adversarial panel of two models from different families, each told to refute what it could, and 618 survived. One note in three (31.3% [27.0, 35.6]) carries a verified failure, concentrated in allergy and medication information, invented patient identity, and history written up as examination on telephone consultations that can contain none. No product was given a patient record; setting aside the two classes a record would have prefilled, invented identity and dates, the rate is 24.8% [20.8, 29.0]. One failure mode did not fit our scheme, drawn from published scribe-error taxonomies: a treatment the clinician retracts, recorded as delivered care. Two clinicians adjudicated blind, disjoint samples: a physician author upheld 20 of 21 findings (95.2% [77.3, 99.2]) and an independent clinician, not an author, 12 of 12 ([75.8, 100]); both judged every sampled refusal genuine. A failure rate depends on the instrument as much as the scribes. With model, evidence and settings fixed, the review instruction alone moves the share of candidates verified from 9.3% to 79.0%, and the reviewing family moves it too: alone at that instruction the gentler flags 54.8% of notes against 27.8%. Between 28% and 97% of sampled notes carry a failure depending on the standard. Published audits disagree among themselves by a margin instrument differences alone can produce: omission is 54-86% of their errors against our 23.1%. We release all 618 findings with transcript-side evidence, every prompt and model version, and the re-runnable pipeline.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 11 of its 50 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

11 of 50 references.

References · 11

  1. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  2. 2022PriMock57: A Dataset Of Primary Care Mock Consultationscited by 31
  3. 2023An Investigation of Evaluation Metrics for Automated Medical Note Generationcited by 10
  4. 2025Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Modelscited by 9
  5. 2022Consultation Checklists: Standardising the Human Evaluation of Medical Note Generationcited by 8
  6. 2023Extrinsically-Focused Evaluation of Omissions in Medical Summarizationcited by 6
  7. 2025From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notescited by 4
  8. 2024DeepScore: A Comprehensive Approach to Measuring Quality in AI-Generated Clinical Documentationcited by 2
  9. 2025FactsR: A Safer Method for Producing High Quality Healthcare Documentationcited by 2
  10. 2026Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluationcited by 2
  11. 2026Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.