# Quantifying Bias in Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2103.15122 · HTML: https://mlsanity.com/paper/2103.15122/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 26 references that are in the map.
- First posted on arXiv 28 Mar 2021.

## Builds on (most cited first, 4 of 4)

- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance](https://mlsanity.com/paper/1908.08717/) (2019)
- [See what I'm saying? Comparing Intelligent Personal Assistant use for Native and Non-Native Language Speakers](https://mlsanity.com/paper/2006.06328/) (2020)

## Built on by (most cited first, 30 of 50)

- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Representation Bias in Data: A Survey on Identification and Resolution Techniques](https://mlsanity.com/paper/2203.11852/) (2022)
- [Can Whisper perform speech-based in-context learning?](https://mlsanity.com/paper/2309.07081/) (2023)
- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models](https://mlsanity.com/paper/2407.06957/) (2024)
- [Bias and Fairness in Chatbots: An Overview](https://mlsanity.com/paper/2309.08836/) (2023)
- [Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Review](https://mlsanity.com/paper/2211.09511/) (2022)
- [Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models](https://mlsanity.com/paper/2408.07665/) (2024)
- [A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systems](https://mlsanity.com/paper/2204.01397/) (2022)
- [Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech](https://mlsanity.com/paper/2405.06150/) (2024)
- [Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes](https://mlsanity.com/paper/2306.04306/) (2023)
- [Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clustering](https://mlsanity.com/paper/2306.06083/) (2023)
- [Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition](https://mlsanity.com/paper/2310.18877/) (2023)
- [FairSSD: Understanding Bias in Synthetic Speech Detectors](https://mlsanity.com/paper/2404.10989/) (2024)
- [The Balancing Act: Unmasking and Alleviating ASR Biases in Portuguese](https://mlsanity.com/paper/2402.07513/) (2024)
- [Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants](https://mlsanity.com/paper/2405.19342/) (2024)
- [Pathological voice adaptation with autoencoder-based voice conversion](https://mlsanity.com/paper/2106.08427/) (2021)
- [Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness](https://mlsanity.com/paper/2211.05809/) (2022)
- [Towards dialect-inclusive recognition in a low-resource language: are balanced corpora the answer?](https://mlsanity.com/paper/2307.07295/) (2023)
- [A Deep Dive into the Disparity of Word Error Rates Across Thousands of NPTEL MOOC Videos](https://mlsanity.com/paper/2307.10587/) (2023)
- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [No Pitch Left Behind: Addressing Gender Unbalance in Automatic Speech Recognition through Pitch Manipulation](https://mlsanity.com/paper/2310.06590/) (2023)
- [Better Transcription of UK Supreme Court Hearings](https://mlsanity.com/paper/2211.17094/) (2022)
- [Debiased Automatic Speech Recognition for Dysarthric Speech via Sample Reweighting with Sample Affinity Test](https://mlsanity.com/paper/2305.13108/) (2023)
- [AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents](https://mlsanity.com/paper/2402.01152/) (2024)
- [DENOASR: Debiasing ASRs through Selective Denoising](https://mlsanity.com/paper/2410.16712/) (2024)
- [SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniors](https://mlsanity.com/paper/2503.16578/) (2025)
- [Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia](https://mlsanity.com/paper/2506.08846/) (2025)
- [Using Data Augmentations and VTLN to Reduce Bias in Dutch End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2307.02009/) (2023)
- [On the Role of Speech Data in Reducing Toxicity Detection Bias](https://mlsanity.com/paper/2411.08135/) (2024)

## Abstract (arXiv)

Automatic speech recognition (ASR) systems promise to deliver objective interpretation of human speech. Practice and recent evidence suggests that the state-of-the-art (SotA) ASRs struggle with the large variation in speech due to e.g., gender, age, speech impairment, race, and accents. Many factors can cause the bias of an ASR system. Our overarching goal is to uncover bias in ASR systems to work towards proactive bias mitigation in ASR. This paper is a first step towards this goal and systematically quantifies the bias of a Dutch SotA ASR system against gender, age, regional accents and non-native accents. Word error rates are compared, and an in-depth phoneme-level error analysis is conducted to understand where bias is occurring. We primarily focus on bias due to articulation differences in the dataset. Based on our findings, we suggest bias mitigation strategies for ASR development.