mlsanityData to 14 Oct 2026

LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT

arXiv 2203.15610 ↗First posted 29 Mar 2022eess.AS cs.CL cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech representation learning has shown promising results in various speech processing tasks. However, the pre-trained models, e.g., HuBERT, are storage-intensive Transformers, limiting their scope of applications under low-resource settings. To this end, we propose LightHuBERT, a once-for-all Transformer compression framework, to find the desired architectures automatically by pruning structured parameters. More precisely, we create a Transformer-based supernet that is nested with thousands of weight-sharing subnets and design a two-stage distillation strategy to leverage the contextualized latent representations from HuBERT. Experiments on automatic speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT enables over 109 architectures concerning the embedding dimension, attention dimension, head number, feed-forward network ratio, and network depth. LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the HuBERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a 3.5× compression ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss. The code and pre-trained models are available at https://github.com/mechanicalsea/lighthubert.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

28 of 33 references.

References · 28

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2019The State of Sparsity in Deep Neural Networkscited by 476
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  11. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
16 more
  1. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  2. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  3. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  4. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  5. 2019A Survey of Deep Learning Applications to Autonomous Vehicle Controlcited by 106
  6. 2019Knowledge Distillation from Internal Representationscited by 87
  7. 2022Unified Visual Transformer Compressioncited by 79
  8. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  9. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  10. 2020Neural Architecture Transfercited by 60
  11. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  12. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  13. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 21
  14. 2019Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networkscited by 17
  15. 2021You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradientcited by 14
  16. 2021EfficientTDNN: Efficient Architecture Search for Speaker Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 21; 2024: 9; 2025: 7; 2026: 2.

50 papers.

Sort:

By area:

  1. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  2. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  3. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  4. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  5. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  6. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  7. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  8. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 25
  9. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  10. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  11. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 15
  12. 2023DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognitioncited by 11
38 more
  1. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  2. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  3. 2024Automatic Speech Recognition with BERT and CTC Transformers: A Reviewcited by 8
  4. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Modelcited by 7
  5. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  6. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  7. 2022Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformerscited by 5
  8. 2024Efficiently Distilling LLMs for Edge Applicationscited by 5
  9. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  10. 2022Continual Learning for On-Device Speech Recognition using Disentangled Conformerscited by 4
  11. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillationcited by 4
  12. 2024Domain Adaptation in Intent Classification Systems: A Reviewcited by 4
  13. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  14. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  15. 2022Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 3
  16. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verificationcited by 3
  17. 2023STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Modelscited by 3
  18. 2025StableQuant: Layer Adaptive Post-Training Quantization for Speech Foundation Modelscited by 3
  19. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objectivecited by 3
  20. 2022Adversarial Speaker Distillation for Countermeasure Model on Automatic Speaker Verificationcited by 2
  21. 2022Once-for-All Sequence Compression for Self-Supervised Speech Modelscited by 2
  22. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  23. 2022Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancementcited by 1
  24. 2023Small-footprint slimmable networks for keyword spottingcited by 1
  25. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  26. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  27. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  28. 2025HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 1
  29. 2022Real-time Speech Interruption Analysis: From Cloud to Client Deploymentcited by 0
  30. 2022Application of Knowledge Distillation to Multi-task Speech Representation Learningcited by 0
  31. 2023Mispronunciation detection using self-supervised speech representationscited by 0
  32. 2023GenDistiller: Distilling Pre-trained Language Models based on Generative Modelscited by 0
  33. 2023Noise robust distillation of self-supervised speech models via correlation metricscited by 0
  34. 2024GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Modelcited by 0
  35. 2024Speech Representation Learning Revisited: The Necessity of Separate Learnable Parameters and Robust Data Augmentationcited by 0
  36. 2025Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Modelscited by 0
  37. 2026HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 0
  38. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0