mlsanityData to 14 Oct 2026

Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition

arXiv 2109.06870 ↗First posted 14 Sep 2021cs.CL cs.LG cs.SDcited by 68

Not on the map. Why?

Abstract (arXiv)

This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition (ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance and its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a pre-trained model architecture with significant improvements along both performance and efficiency dimensions across a variety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x inference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference time, SEW reduces word error rate by 25-50% across different model sizes.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 50 of its 75 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

50 of 75 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2018Self-Attention with Relative Position Representationscited by 1,120
  8. 2016Deep Networks with Stochastic Depthcited by 979
  9. 2016Language Modeling with Gated Convolutional Networkscited by 702
  10. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  11. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  12. 2015Listen, Attend and Spellcited by 543
  13. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  14. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  15. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  16. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  17. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  18. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  19. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  20. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  21. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  22. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  23. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  24. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  25. 2014A* Samplingcited by 151
  26. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  27. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  28. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  29. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  30. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 83
  31. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  32. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 69
  33. 2021Fast and Accurate Model Scalingcited by 46
  34. 2016Wav2Letter: an End-to-End ConvNet-based Speech Recognition Systemcited by 43
  35. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  36. 2018Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spacescited by 34
  37. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 32
  38. 2019Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networkscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 36; 2023: 17; 2024: 4; 2025: 3; 2026: 1.

68 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  4. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  5. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  6. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  7. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  8. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  9. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  10. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 87
  11. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  12. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
38 more
  1. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  2. 2022Audio Self-supervised Learning: A Surveycited by 65
  3. 2022Contrastive language and vision learning of general fashion conceptscited by 42
  4. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  5. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  6. 2023AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASRcited by 35
  7. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  8. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  9. 2023Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalizationcited by 28
  10. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 27
  11. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  12. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  13. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  14. 2023Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretrainingcited by 18
  15. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  16. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  17. 2023SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processingcited by 16
  18. 2023Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognitioncited by 16
  19. 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
  20. 2022On-demand compute reduction with stochastic wav2vec 2.0cited by 10
  21. 2022CAB: Comprehensive Attention Benchmarking on Long Sequence Modelingcited by 10
  22. 2022Wav2Vec-Aug: Improved self-supervised training with limited datacited by 9
  23. 2022A Comparative Study of Self-supervised Speech Representation Based Voice Conversioncited by 9
  24. 2022Self-supervised Graphs for Audio Representation Learning with Limited Labeled Datacited by 8
  25. 2022On Compressing Sequences for Self-Supervised Speech Modelscited by 8
  26. 2022Neural Network based Formation of Cognitive Maps of Semantic Spaces and the Emergence of Abstract Conceptscited by 8
  27. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
  28. 2022The Efficacy of Self-Supervised Speech Models for Audio Representationscited by 6
  29. 2022End-to-End and Self-Supervised Learning for ComParE 2022 Stuttering Sub-Challengecited by 5
  30. 2022Efficient Speech Representation Learning with Low-Bit Quantizationcited by 5
  31. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  32. 2021K-Wav2vec 2.0: Automatic Speech Recognition based on Joint Decoding of Graphemes and Syllablescited by 4
  33. 2022T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5cited by 4
  34. 2022Learning General Audio Representations with Large-Scale Training of Patchout Audio Transformerscited by 4
  35. 2023Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representationscited by 4
  36. 2023Comparative Analysis of the wav2vec 2.0 Feature Extractorcited by 4
  37. 2024A Backdoor Approach with Inverted Labels Using Dirty Label-Flipping Attackscited by 4
  38. 2024Sustainable self-supervised learning for speech representationscited by 4

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model
  2. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprint
  3. 2025Training and Inference Efficiency of Encoder-Decoder Speech Models
  4. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  5. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  6. 2025Quantizing Whisper-small: How design choices affect ASR performance
  7. 2025Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models
  8. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs