# From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation

arXiv: https://arxiv.org/abs/2604.14152 · HTML: https://mlsanity.com/paper/2604.14152/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 19 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 6 of 6)

- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [MediaSpeech: Multilanguage ASR Benchmark and Dataset](https://mlsanity.com/paper/2103.16193/) (2021)
- [Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech](https://mlsanity.com/paper/2405.06150/) (2024)
- [Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings](https://mlsanity.com/paper/2303.05737/) (2023)
- [Leveraging Imperfection with MEDLEY A Multi-Model Approach Harnessing Bias in Medical AI](https://mlsanity.com/paper/2508.21648/) (2025)
- [ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages](https://mlsanity.com/paper/2512.10967/) (2025)

## Abstract (arXiv)

Ambient AI "scribe" systems promise to reduce clinical documentation burden, but automatic speech recognition (ASR) errors can remain unnoticed without careful review, and high-quality human reference transcripts are often unavailable for calibrating uncertainty. We investigate whether cross-model disagreement among heterogeneous ASR systems can act as a reference-free uncertainty signal to prioritize human verification in medical transcription workflows. Using 50 publicly available medical education audio clips (8 h 14 min), we transcribed each clip with eight ASR systems spanning commercial APIs and open-source engines. We aligned multi-model outputs, built consensus pseudo-references, and quantified token-level agreement using a majority-strength metric; we further characterized disagreements by type (content vs. punctuation/formatting) and assessed per-model agreement via leave-one-model-out (jackknife) consensus scoring. Inter-model reliability was low (ICC[2,1] = 0.131), indicating heterogeneous failure modes across systems. Across 76,398 evaluated token positions, 72.1% showed near-unanimous agreement (7-8 models), while 2.5% fell into high-risk bands (0-3 models), with high-risk mass varying from 0.7% to 11.4% across accent groups. Low-agreement regions were enriched for content disagreements, with the content fraction increasing from 53.9% to 73.9% across quintiles of high-risk mass. These results suggest that cross-model disagreement provides a sparse, localizable signal that can surface potentially unreliable transcript spans without human-verified references, enabling targeted review; clinical accuracy of flagged regions remains to be established.