mlsanityData to 7 Oct 2026

Don't speak too fast: The impact of data bias on self-supervised speech models

arXiv 2110.07957 ↗First posted 15 Oct 2021eess.AS cs.CL cs.SDcited by 20

Not on the map. Why?

Abstract (arXiv)

Self-supervised Speech Models (S3Ms) have been proven successful in many speech downstream tasks, like ASR. However, how pre-training data affects S3Ms' downstream behavior remains an unexplored issue. In this paper, we study how pre-training data affects S3Ms by pre-training models on biased datasets targeting different factors of speech, including gender, content, and prosody, and evaluate these pre-trained S3Ms on selected downstream tasks in SUPERB Benchmark. Our experiments show that S3Ms have tolerance toward gender bias. Moreover, we find that the content of speech has little impact on the performance of S3Ms across downstream tasks, but S3Ms do show a preference toward a slower speech rate.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

22 of 37 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Deep contextualized word representationscited by 2,285
  3. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  5. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  6. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  7. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  8. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  9. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  10. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  11. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  12. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
10 more
  1. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  2. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  3. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  4. 2020Improved Speech Representations with Multi-Target Autoregressive Predictive Codingcited by 23
  5. 2020Breeding Gender-aware Direct Speech Translation Systemscited by 16
  6. 2020Understanding Self-Attention of Self-Supervised Audio Transformerscited by 15
  7. 2020A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognitioncited by 12
  8. 2021How to Split: the Effect of Word Segmentation on Gender Bias in Speech Translationcited by 11
  9. 2021Improving Fairness in Speaker Recognitioncited by 5
  10. 2020Masked Pre-trained Encoder base on Joint CTC-Transformercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 6; 2025: 5; 2026: 3.

20 papers.

Sort:

  1. 2022How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communicationscited by 30
  2. 2024Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Modelscited by 20
  3. 2024On the social bias of speech self-supervised modelscited by 16
  4. 2024Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Modelscited by 15
  5. 2022A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systemscited by 14
  6. 2023Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognitioncited by 9
  7. 2024Self-supervised Speech Representations Still Struggle with African American Vernacular Englishcited by 6
  8. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  9. 2025Mitigating Subgroup Disparities in Multi-Label Speech Emotion Recognition: A Pseudo-Labeling and Unsupervised Learning Approachcited by 5
  10. 2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
  11. 2023Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Modelscited by 3
  12. 2023Some voices are too common: Building fair speech recognition systems using the Common Voice datasetcited by 2
8 more
  1. 2024On the Role of Speech Data in Reducing Toxicity Detection Biascited by 2
  2. 2025Exploring Gender Disparities in Automatic Speech Recognition Technologycited by 2
  3. 2025CO-VADA: A Confidence-Oriented Voice Augmentation Debiasing Approach for Fair Speech Emotion Recognitioncited by 1
  4. 2026Where Do Self-Supervised Speech Models Become Unfair?cited by 1
  5. 2026Responsible Benchmarking of Fairness for Automatic Speech Recognitioncited by 1
  6. 2023Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancementcited by 0
  7. 2025Self-supervised learning of speech representations with Dutch archival datacited by 0
  8. 2026"OK Aura, Be Fair With Me": Demographics-Agnostic Training for Bias Mitigation in Wake-up Word Detectioncited by 0