mlsanityData to 14 Oct 2026

Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions

arXiv 2111.09983 ↗First posted 18 Nov 2021eess.AS cs.SDcited by 26

Not on the map. Why?

Abstract (arXiv)

It is well known that many machine learning systems demonstrate bias towards specific groups of individuals. This problem has been studied extensively in the Facial Recognition area, but much less so in Automatic Speech Recognition (ASR). This paper presents initial Speech Recognition results on "Casual Conversations" -- a publicly released 846 hour corpus designed to help researchers evaluate their computer vision and audio models for accuracy across a diverse set of metadata, including age, gender, and skin tone. The entire corpus has been manually transcribed, allowing for detailed ASR evaluations across these metadata. Multiple ASR models are evaluated, including models trained on LibriSpeech, 14,000 hour transcribed, and over 2 million hour untranscribed social media videos. Significant differences in word error rate across gender and skin tone are observed at times for all models. We are releasing human transcripts from the Casual Conversations dataset to encourage the community to develop a variety of techniques to reduce these statistical biases.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Nov 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 6; 2024: 7; 2025: 4; 2026: 3.

26 papers.

Sort:

By area:

  1. 2022Representation Bias in Data: A Survey on Identification and Resolution Techniquescited by 37
  2. 2022Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparitiescited by 28
  3. 2024Towards measuring fairness in speech recognition: Fair-Speech datasetcited by 26
  4. 2024Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gapscited by 25
  5. 2023Consensus and Subjectivity of Skin Tone Annotation for ML Fairnesscited by 22
  6. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 21
  7. 2022Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Reviewcited by 19
  8. 2024Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speechcited by 15
  9. 2022A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairnesscited by 13
  10. 2023Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clusteringcited by 11
  11. 2024A Contrastive Learning Approach to Mitigate Bias in Speech Modelscited by 9
  12. 2025Automatic Speech Recognition Biases in Newcastle English: an Error Analysiscited by 6
14 more
  1. 2022Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustnesscited by 5
  2. 2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
  3. 2023Some voices are too common: Building fair speech recognition systems using the Common Voice datasetcited by 3
  4. 2025How to Evaluate Automatic Speech Recognition: Comparing Different Performance and Bias Measurescited by 3
  5. 2023Using Data Augmentations and VTLN to Reduce Bias in Dutch End-to-End Speech Recognition Systemscited by 2
  6. 2025Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Modelscited by 2
  7. 2021Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speechcited by 1
  8. 2023Synthetic Cross-accent Data Augmentation for Automatic Speech Recognitioncited by 1
  9. 2023Elucidate Gender Fairness in Singing Voice Transcriptioncited by 0
  10. 2024FairLENS: Assessing Fairness in Law Enforcement Speech Recognitioncited by 0
  11. 2024Everyone deserves their voice to be heard: Analyzing Predictive Gender Bias in ASR Models Applied to Dutch Speech Datacited by 0
  12. 2026"This Wasn't Made for Me": Recentering User Experience and Emotional Impact in the Evaluation of ASR Biascited by 0
  13. 2026Queer inclusion in speech datasets: An audit and taxonomy of practical tensionscited by 0
  14. 2026When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Modelscited by 0