# Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech arXiv: https://arxiv.org/abs/2405.06150 ยท HTML: https://mlsanity.com/paper/2405.06150/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 61 references that are in the map. - First posted on arXiv 10 May 2024. ## Builds on (most cited first, 22 of 22) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017) - [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017) - [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021) - [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023) - [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021) - [Quantifying Bias in Automatic Speech Recognition](https://mlsanity.com/paper/2103.15122/) (2021) - [Personalizing ASR for Dysarthric and Accented Speech with Limited Data](https://mlsanity.com/paper/1907.13511/) (2019) - [Disfluency Detection using a Bidirectional LSTM](https://mlsanity.com/paper/1604.03209/) (2016) - [Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions](https://mlsanity.com/paper/2111.09983/) (2021) - [From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition](https://mlsanity.com/paper/2302.09044/) (2023) - [Artificial Intelligence and Life in 2030: The One Hundred Year Study on Artificial Intelligence](https://mlsanity.com/paper/2211.06318/) (2022) - [Personalized Automatic Speech Recognition Trained on Small Disordered Speech Datasets](https://mlsanity.com/paper/2110.04612/) (2021) - [Analysis and Tuning of a Voice Assistant System for Dysfluent Speech](https://mlsanity.com/paper/2106.11759/) (2021) - [Enhancing ASR for Stuttered Speech with Limited Data Using Detect and Pass](https://mlsanity.com/paper/2202.05396/) (2022) - [FluentNet: End-to-End Detection of Speech Disfluency with Deep Learning](https://mlsanity.com/paper/2009.11394/) (2020) - [Assessing ASR Model Quality on Disordered Speech using BERTScore](https://mlsanity.com/paper/2209.10591/) (2022) - [Person Perception Biases Exposed: Revisiting the First Impressions Dataset](https://mlsanity.com/paper/2011.14906/) (2020) ## Built on by (most cited first, 15 of 15) - [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025) - [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025) - [Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models](https://mlsanity.com/paper/2606.14647/) (2026) - [Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia](https://mlsanity.com/paper/2506.08846/) (2025) - [Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbones](https://mlsanity.com/paper/2509.20321/) (2025) - [Talking to Robots: A Practical Examination of Speech Foundation Models for HRI Applications](https://mlsanity.com/paper/2508.17753/) (2025) - [Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR](https://mlsanity.com/paper/2510.10738/) (2025) - [Unheard in the Digital Age: Rethinking AI Bias and Speech Diversity](https://mlsanity.com/paper/2601.18641/) (2026) - ["OK Aura, Be Fair With Me": Demographics-Agnostic Training for Bias Mitigation in Wake-up Word Detection](https://mlsanity.com/paper/2604.05830/) (2026) - [From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation](https://mlsanity.com/paper/2604.14152/) (2026) - [Aligning Stuttered-Speech Research with End-User Needs: Scoping Review, Survey, and Guidelines](https://mlsanity.com/paper/2604.20535/) (2026) - [Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR](https://mlsanity.com/paper/2606.14391/) (2026) - [MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition](https://mlsanity.com/paper/2606.14459/) (2026) - [What Counts as an Error? Dual-Reference Benchmarking for Atypical ASR](https://mlsanity.com/paper/2606.31112/) (2026) - [Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation](https://mlsanity.com/paper/2609.38976/) (2026) ## Abstract (arXiv) Automatic speech recognition (ASR) systems, increasingly prevalent in education, healthcare, employment, and mobile technology, face significant challenges in inclusivity, particularly for the 80 million-strong global community of people who stutter. These systems often fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations. This study evaluates six leading ASRs, analyzing their performance on both a real-world dataset of speech samples from individuals who stutter and a synthetic dataset derived from the widely-used LibriSpeech benchmark. The synthetic dataset, uniquely designed to incorporate various stuttering events, enables an in-depth analysis of each ASR's handling of disfluent speech. Our comprehensive assessment includes metrics such as word error rate (WER), character error rate (CER), and semantic accuracy of the transcripts. The results reveal a consistent and statistically significant accuracy bias across all ASRs against disfluent speech, manifesting in significant syntactical and semantic inaccuracies in transcriptions. These findings highlight a critical gap in current ASR technologies, underscoring the need for effective bias mitigation strategies. Addressing this bias is imperative not only to improve the technology's usability for people who stutter but also to ensure their equitable and inclusive participation in the rapidly evolving digital landscape.