mlsanityData to 14 Oct 2026

Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute

arXiv 2306.06672 ↗First posted 11 Jun 2023cs.CL cs.AI eess.AScited by 21

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has led to great strides in speech processing. However, the resources needed to train these models has become prohibitively large as they continue to scale. Currently, only a few groups with substantial resources are capable of creating SSL models, which harms reproducibility. In this work, we optimize HuBERT SSL to fit in academic constraints. We reproduce HuBERT independently from the original implementation, with no performance loss. Our code and training optimizations make SSL feasible with only 8 GPUs, instead of the 32 used in the original work. We also explore a semi-supervised route, using an ASR model to skip the first pre-training iteration. Within one iteration of pre-training, our models improve over HuBERT on several tasks. Furthermore, our HuBERT Large variant requires only 8 GPUs, achieving similar performance to the original trained on 128. As our contribution to the community, all models, configurations, and code are made open-source in ESPnet.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
20 more
  1. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  3. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  5. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  6. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  8. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  9. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  10. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  11. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  12. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  13. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  14. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  15. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  16. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 31
  17. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  18. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  19. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  20. 2022Biased Self-supervised learning for ASRcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 10; 2025: 4; 2026: 1.

21 papers.

Sort:

  1. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  2. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  3. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 50
  4. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 18
  5. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  6. 2023Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyondcited by 10
  7. 2024Open Implementation and Study of BEST-RQ for Speech Processingcited by 9
  8. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  9. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  10. 2023Evaluating Self-Supervised Speech Representations for Indigenous American Languagescited by 6
  11. 2024The Effect of Batch Size on Contrastive Self-Supervised Speech Representation Learningcited by 5
  12. 2025Towards Early Prediction of Self-Supervised Speech Model Performancecited by 5
9 more
  1. 2024Sustainable self-supervised learning for speech representationscited by 4
  2. 2025VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretrainingcited by 3
  3. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  4. 2024Linear-Complexity Self-Supervised Learning for Speech Processingcited by 2
  5. 2024An Analysis of Linear Complexity Attention Substitutes with BEST-RQcited by 2
  6. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  7. 2024A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancementcited by 1
  8. 2024ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integrationcited by 0
  9. 2026Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokenscited by 0