# Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions

arXiv: https://arxiv.org/abs/2111.09983 · HTML: https://mlsanity.com/paper/2111.09983/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2021.

## Built on by (most cited first, 26 of 26)

- [Representation Bias in Data: A Survey on Identification and Resolution Techniques](https://mlsanity.com/paper/2203.11852/) (2022)
- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [Towards measuring fairness in speech recognition: Fair-Speech dataset](https://mlsanity.com/paper/2408.12734/) (2024)
- [Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps](https://mlsanity.com/paper/2402.17954/) (2024)
- [Consensus and Subjectivity of Skin Tone Annotation for ML Fairness](https://mlsanity.com/paper/2305.09073/) (2023)
- [Speech Robust Bench: A Robustness Benchmark For Speech Recognition](https://mlsanity.com/paper/2403.07937/) (2024)
- [Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Review](https://mlsanity.com/paper/2211.09511/) (2022)
- [Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech](https://mlsanity.com/paper/2405.06150/) (2024)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clustering](https://mlsanity.com/paper/2306.06083/) (2023)
- [A Contrastive Learning Approach to Mitigate Bias in Speech Models](https://mlsanity.com/paper/2406.14686/) (2024)
- [Automatic Speech Recognition Biases in Newcastle English: an Error Analysis](https://mlsanity.com/paper/2506.16558/) (2025)
- [Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness](https://mlsanity.com/paper/2211.05809/) (2022)
- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [Some voices are too common: Building fair speech recognition systems using the Common Voice dataset](https://mlsanity.com/paper/2306.03773/) (2023)
- [How to Evaluate Automatic Speech Recognition: Comparing Different Performance and Bias Measures](https://mlsanity.com/paper/2507.05885/) (2025)
- [Using Data Augmentations and VTLN to Reduce Bias in Dutch End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2307.02009/) (2023)
- [Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models](https://mlsanity.com/paper/2504.15093/) (2025)
- [Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech](https://mlsanity.com/paper/2112.13758/) (2021)
- [Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition](https://mlsanity.com/paper/2303.00802/) (2023)
- [Elucidate Gender Fairness in Singing Voice Transcription](https://mlsanity.com/paper/2308.02898/) (2023)
- [FairLENS: Assessing Fairness in Law Enforcement Speech Recognition](https://mlsanity.com/paper/2405.13166/) (2024)
- [Everyone deserves their voice to be heard: Analyzing Predictive Gender Bias in ASR Models Applied to Dutch Speech Data](https://mlsanity.com/paper/2411.09431/) (2024)
- ["This Wasn't Made for Me": Recentering User Experience and Emotional Impact in the Evaluation of ASR Bias](https://mlsanity.com/paper/2604.21148/) (2026)
- [Queer inclusion in speech datasets: An audit and taxonomy of practical tensions](https://mlsanity.com/paper/2609.25491/) (2026)
- [When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models](https://mlsanity.com/paper/2609.27382/) (2026)

## Abstract (arXiv)

It is well known that many machine learning systems demonstrate bias towards specific groups of individuals. This problem has been studied extensively in the Facial Recognition area, but much less so in Automatic Speech Recognition (ASR). This paper presents initial Speech Recognition results on "Casual Conversations" -- a publicly released 846 hour corpus designed to help researchers evaluate their computer vision and audio models for accuracy across a diverse set of metadata, including age, gender, and skin tone. The entire corpus has been manually transcribed, allowing for detailed ASR evaluations across these metadata. Multiple ASR models are evaluated, including models trained on LibriSpeech, 14,000 hour transcribed, and over 2 million hour untranscribed social media videos. Significant differences in word error rate across gender and skin tone are observed at times for all models. We are releasing human transcripts from the Casual Conversations dataset to encourage the community to develop a variety of techniques to reduce these statistical biases.