mlsanityData to 7 Oct 2026

Shrinking Bigfoot: Reducing wav2vec 2.0 footprint

arXiv 2103.15760 ↗First posted 29 Mar 2021cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Wav2vec 2.0 is a state-of-the-art speech recognition model which maps speech audio waveforms into latent representations. The largest version of wav2vec 2.0 contains 317 million parameters. Hence, the inference latency of wav2vec 2.0 will be a bottleneck in production, leading to high costs and a significant environmental footprint. To improve wav2vec's applicability to a production setting, we explore multiple model compression methods borrowed from the domain of large language models. Using a teacher-student approach, we distilled the knowledge from the original wav2vec 2.0 model into a student model, which is 2 times faster and 4.8 times smaller than the original model. This increase in performance is accomplished with only a 7% degradation in word error rate (WER). Our quantized model is 3.6 times smaller than the original model, with only a 0.1% degradation in WER. To the best of our knowledge, this is the first work that compresses wav2vec 2.0.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

23 of 38 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
11 more
  1. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  2. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  3. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  4. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 168
  5. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  6. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 38
  7. 2019On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compressioncited by 14
  8. 2019Neural Network Distiller: A Python Package For DNN Compression Researchcited by 12
  9. 2020Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modelingcited by 9
  10. 2020Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognitioncited by 6
  11. 2020Iterative Compression of End-to-End ASR Model using AutoMLcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 3; 2024: 3; 2025: 1; 2026: 1.

16 papers.

Sort:

By area:

  1. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  2. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  3. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  4. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  5. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  6. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  7. 2022Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformerscited by 5
  8. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  9. 2022Efficient Utilization of Large Pre-Trained Models for Low Resource ASRcited by 4
  10. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  11. 2024AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weightingcited by 3
  12. 2022RedApt: An Adaptor for wav2vec 2 Encoding Faster and Smaller Speech Translation without Quality Compromisecited by 2
4 more
  1. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  2. 2022Application of Knowledge Distillation to Multi-task Speech Representation Learningcited by 0
  3. 2024AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformerscited by 0
  4. 2026Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Modelscited by 0