# Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps

arXiv: https://arxiv.org/abs/2402.17954 · HTML: https://mlsanity.com/paper/2402.17954/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 112 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020)
- [What's In My Big Data?](https://mlsanity.com/paper/2310.20707/) (2023)
- [VoxLingua107: a Dataset for Spoken Language Recognition](https://mlsanity.com/paper/2011.12998/) (2020)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks](https://mlsanity.com/paper/2210.02938/) (2022)
- [Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation](https://mlsanity.com/paper/2402.12649/) (2024)
- [Men Are Elected, Women Are Married: Events Gender Bias on Wikipedia](https://mlsanity.com/paper/2106.01601/) (2021)
- [Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions](https://mlsanity.com/paper/2111.09983/) (2021)

## Built on by (most cited first, 25 of 25)

- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- ["It's not a representation of me": Examining Accent Bias and Digital Exclusion in Synthetic AI Voice Services](https://mlsanity.com/paper/2504.09346/) (2025)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation](https://mlsanity.com/paper/2411.01710/) (2024)
- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [On the Encoding of Gender in Transformer-based ASR Representations](https://mlsanity.com/paper/2406.09855/) (2024)
- [Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations](https://mlsanity.com/paper/2602.01030/) (2026)
- [Different Speech Translation Models Encode and Translate Speaker Gender Differently](https://mlsanity.com/paper/2506.02172/) (2025)
- [Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia](https://mlsanity.com/paper/2506.08846/) (2025)
- [FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian](https://mlsanity.com/paper/2505.22759/) (2025)
- [Echoes of Phonetics: Unveiling Relevant Acoustic Cues for ASR via Feature Attribution](https://mlsanity.com/paper/2506.02181/) (2025)
- [Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models](https://mlsanity.com/paper/2604.22631/) (2026)
- [Do What I Say: A Spoken Prompt Dataset for Instruction-Following](https://mlsanity.com/paper/2603.09881/) (2026)
- [Greater accessibility can amplify discrimination in generative AI](https://mlsanity.com/paper/2603.22260/) (2026)
- [Where Do Self-Supervised Speech Models Become Unfair?](https://mlsanity.com/paper/2604.18249/) (2026)
- [Responsible Benchmarking of Fairness for Automatic Speech Recognition](https://mlsanity.com/paper/2605.10615/) (2026)
- [Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning](https://mlsanity.com/paper/2506.17019/) (2025)
- ["OK Aura, Be Fair With Me": Demographics-Agnostic Training for Bias Mitigation in Wake-up Word Detection](https://mlsanity.com/paper/2604.05830/) (2026)
- [Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios](https://mlsanity.com/paper/2606.06177/) (2026)
- [Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems](https://mlsanity.com/paper/2608.30853/) (2026)
- [Nuha-Speech: Building General-Purpose Arabic Speech-LLMs](https://mlsanity.com/paper/2609.11892/) (2026)
- [A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models](https://mlsanity.com/paper/2609.18533/) (2026)
- [Queer inclusion in speech datasets: An audit and taxonomy of practical tensions](https://mlsanity.com/paper/2609.25491/) (2026)
- [Towards participatory speech dataset curation: A queer case study and conceptual framework](https://mlsanity.com/paper/2609.25496/) (2026)
- [Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?](https://mlsanity.com/paper/2610.13550/) (2026)

## Abstract (arXiv)

Current automatic speech recognition (ASR) models are designed to be used across many languages and tasks without substantial changes. However, this broad language coverage hides performance gaps within languages, for example, across genders. Our study systematically evaluates the performance of two widely used multilingual ASR models on three datasets, encompassing 19 languages from eight language families and two speaking conditions. Our findings reveal clear gender disparities, with the advantaged group varying across languages and models. Surprisingly, those gaps are not explained by acoustic or lexical properties. However, probing internal model states reveals a correlation with gendered performance gap. That is, the easier it is to distinguish speaker gender in a language using probes, the more the gap reduces, favoring female speakers. Our results show that gender disparities persist even in state-of-the-art models. Our findings have implications for the improvement of multilingual ASR systems, underscoring the importance of accessibility to training data and nuanced evaluation to predict and mitigate gender gaps. We release all code and artifacts at https://github.com/g8a9/multilingual-asr-gender-gap.