mlsanityData to 7 Oct 2026

Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers

arXiv 2211.09949 ↗First posted 17 Nov 2022cs.CL cs.LG cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

Transformer-based self-supervised models have achieved remarkable success in speech processing, but their large size and high inference cost present significant challenges for real-world deployment. While numerous compression techniques have been proposed, inconsistent evaluation metrics make it difficult to compare their practical effectiveness. In this work, we conduct a comprehensive study of four common compression methods, including weight pruning, head pruning, low-rank approximation, and knowledge distillation on self-supervised speech Transformers. We evaluate each method under three key metrics: parameter count, multiply-accumulate operations, and real-time factor. Results show that each method offers distinct advantages. In addition, we contextualize recent compression techniques, comparing DistilHuBERT, FitHuBERT, LightHuBERT, ARMHuBERT, and STaRHuBERT under the same framework, offering practical guidance on compression for deployment.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

22 of 36 references.

References · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2019Are Sixteen Heads Really Better than One?cited by 803
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2013Do Deep Nets Really Need to be Deep?cited by 466
10 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  2. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  3. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  4. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  5. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  6. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  7. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  8. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  9. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  10. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 0; 2026: 1.

5 papers.

  1. 2023Transformers in Speech Processing: A Surveycited by 38
  2. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  3. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  4. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  5. 2026Fast Speech Foundation Model Distillation Using Interleaved Stackingcited by 0