mlsanityData to 14 Oct 2026

Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models

arXiv 2604.22631 ↗First posted 24 Apr 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Modern automatic speech recognition (ASR) systems have been observed to function better for certain speaker groups (SGs) than others, despite recent gains in overall performance. One potential impediment to progress towards fairer ASR is a more nuanced understanding of the types of modeling errors that speech encoder models make, and in particular the difference between the structure of embeddings for high-performance and low-performance SGs. This paper proposes a framework typifying two types of error that can occur in modeling phonemes in ASR systems: random error/high variance in phoneme embedding, vs systematic error/embedding bias. We find that training phoneme classification probes only on a single, typically disadvantaged SG, sometimes improves performance for that SG, which is evidence for the existence of SG-level bias in phoneme embeddings. On the other hand, we find that speakers and SGs with higher levels of phoneme variance are the same as those with worse phoneme prediction accuracy. We conclude that both types of error are present in phoneme embeddings and both are candidate causes for SG-level unfairness in ASR, though random error is likely a greater hindrance to fairness than systematic error. Furthermore, we find that finetuning encoder models using a fairness-enhancing algorithm (domain enhancing and adversarial training) changes neither the benefits of in-domain phoneme classification probe training, nor measured levels of random embedding error.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

35 of 57 references.

References · 35

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  4. 2020Supervised Contrastive Learningcited by 2,829
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  9. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  10. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  11. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  12. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
23 more
  1. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  2. 2022Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluationcited by 59
  3. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
  4. 2021Quantifying Bias in Automatic Speech Recognitioncited by 50
  5. 2018Domain Adversarial Training for Accented Speech Recognitioncited by 40
  6. 2025Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audiocited by 31
  7. 2022Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparitiescited by 28
  8. 2024Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Modelscited by 27
  9. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  10. 2024Towards measuring fairness in speech recognition: Fair-Speech datasetcited by 26
  11. 2024Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gapscited by 25
  12. 2023Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspacescited by 20
  13. 2021Best of Both Worlds: Robust Accented Speech Recognition with Adversarial Transfer Learningcited by 18
  14. 2025A Preliminary Exploration with GPT-4o Voice Modecited by 16
  15. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  16. 2021Accent-Robust Automatic Speech Recognition Using Supervised and Unsupervised Wav2vec Embeddingscited by 13
  17. 2023Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Featurescited by 7
  18. 2022Enhancing and Adversarial: Improve ASR with Speaker Labelscited by 6
  19. 2024Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistantscited by 6
  20. 2024Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximizationcited by 6
  21. 2024Phoneme-Level Contrastive Learning for User-Defined Keyword Spotting with Flexible Enrollmentcited by 5
  22. 2026A Study of Data Selection Strategies for Pre-training Self-Supervised Speech Modelscited by 3
  23. 2026Where Do Self-Supervised Speech Models Become Unfair?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Modelscited by 0
  2. 2026Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?cited by 0