Beyond Single Ground Truth: Reference Monism as Epistemic Injustice in ASR Evaluation
Not on the map. Why?
Abstract (arXiv)
Automatic speech recognition (ASR) evaluation compares system output to ground truth transcripts, with Word Error Rate (WER) quantifying the distance between them. But ground truth transcripts are not discovered - they are produced by human annotators following conventions that encode normative assumptions about which speech features matter. Different conventions (verbatim, non-verbatim, legal) produce different transcripts of identical speech and judge the same ASR output differently. This paper argues that reference monism - enforcing a single transcription convention as ground truth - commits epistemic injustice. Speakers with aphasia, whose speech includes clinically meaningful disfluencies, are systematically disadvantaged when evaluated against "clean" references that treat those disfluencies as errors. The harm is not merely differential performance, but that evaluative infrastructure lacks interpretive resources to recognize their contributions as legitimate. We develop a philosophical framework introducing the hermeneutical gap, formalize Epistemic Injustice Distance (EID) to measure reference monism's cost, and demonstrate empirically using AphasiaBank that WER varies depending on which convention defines ground truth. We propose WER-Range: reporting performance across legitimate conventions rather than assuming a single correct answer.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 5 of its 73 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
5 of 73 references.
References · 5
- 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 52
- 2024Disclosure and Mitigation of Gender Bias in LLMscited by 49
- 2022Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparitiescited by 28
- 2024Style-agnostic evaluation of ASR using multiple reference transcriptscited by 5
- 2025Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasiacited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
- 2026The Role of Disfluencies in Speech Translationcited by 0
- 2026A Cross Community Agenda for Speech AIcited by 0
- 2026The Role of Disfluencies in Speech Translation3 Aug 2026
- 2026A Cross Community Agenda for Speech AI22 Jul 2026