# Towards measuring fairness in speech recognition: Fair-Speech dataset

arXiv: https://arxiv.org/abs/2408.12734 · HTML: https://mlsanity.com/paper/2408.12734/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 9 of 9)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions](https://mlsanity.com/paper/2111.09983/) (2021)
- [ASR4REAL: An extended benchmark for speech models](https://mlsanity.com/paper/2110.08583/) (2021)
- [Model-Based Approach for Measuring the Fairness in ASR](https://mlsanity.com/paper/2109.09061/) (2021)
- [Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clustering](https://mlsanity.com/paper/2306.06083/) (2023)

## Built on by (most cited first, 26 of 26)

- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [Mind the Gap! Static and Interactive Evaluations of Large Audio Models](https://mlsanity.com/paper/2502.15919/) (2025)
- [Evaluating Bias in Spoken Dialogue LLMs for Real-World Decisions and Recommendations](https://mlsanity.com/paper/2510.02352/) (2025)
- [Voxlect: A Speech Foundation Model Benchmark for Modeling Dialects and Regional Languages Around the Globe](https://mlsanity.com/paper/2508.01691/) (2025)
- [ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems](https://mlsanity.com/paper/2505.11572/) (2025)
- [The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties](https://mlsanity.com/paper/2509.07139/) (2025)
- [DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding](https://mlsanity.com/paper/2601.23161/) (2026)
- [WavShape: Information-Theoretic Speech Representation Learning for Fair and Privacy-Aware Audio Processing](https://mlsanity.com/paper/2506.22789/) (2025)
- [Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models](https://mlsanity.com/paper/2604.22631/) (2026)
- [Fairness in Dysarthric Speech Synthesis: Understanding Intrinsic Bias in Dysarthric Speech Cloning using F5-TTS](https://mlsanity.com/paper/2508.05102/) (2025)
- [Acoustic-based Gender Differentiation in Speech-aware Language Models](https://mlsanity.com/paper/2509.21125/) (2025)
- [Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography](https://mlsanity.com/paper/2603.05267/) (2026)
- [Where Do Self-Supervised Speech Models Become Unfair?](https://mlsanity.com/paper/2604.18249/) (2026)
- [Towards Fair ASR For Second Language Speakers Using Fairness Prompted Finetuning](https://mlsanity.com/paper/2510.18374/) (2025)
- [MedVoiceBias: A Controlled Study of Audio LLM Behavior in Clinical Decision-Making](https://mlsanity.com/paper/2511.06592/) (2025)
- ["OK Aura, Be Fair With Me": Demographics-Agnostic Training for Bias Mitigation in Wake-up Word Detection](https://mlsanity.com/paper/2604.05830/) (2026)
- [Cross-Cultural Bias in Mel-Scale Representations: Evidence and Alternatives from Speech and Music](https://mlsanity.com/paper/2604.10503/) (2026)
- [Premover: Fast Vision-Language-Action Control by Acting Before Instructions Are Complete](https://mlsanity.com/paper/2605.12160/) (2026)
- [Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text](https://mlsanity.com/paper/2608.22908/) (2026)
- [Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems](https://mlsanity.com/paper/2608.30853/) (2026)
- [Queer inclusion in speech datasets: An audit and taxonomy of practical tensions](https://mlsanity.com/paper/2609.25491/) (2026)
- [Towards participatory speech dataset curation: A queer case study and conceptual framework](https://mlsanity.com/paper/2609.25496/) (2026)
- [When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models](https://mlsanity.com/paper/2609.27382/) (2026)
- [Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation](https://mlsanity.com/paper/2609.38976/) (2026)

## Abstract (arXiv)

The current public datasets for speech recognition (ASR) tend not to focus specifically on the fairness aspect, such as performance across different demographic groups. This paper introduces a novel dataset, Fair-Speech, a publicly released corpus to help researchers evaluate their ASR models for accuracy across a diverse set of self-reported demographic information, such as age, gender, ethnicity, geographic variation and whether the participants consider themselves native English speakers. Our dataset includes approximately 26.5K utterances in recorded speech by 593 people in the United States, who were paid to record and submit audios of themselves saying voice commands. We also provide ASR baselines, including on models trained on transcribed and untranscribed social media videos and open source models.