mlsanityData to 7 Oct 2026

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

arXiv 2106.07447 ↗First posted 14 Jun 2021cs.CL cs.AI cs.LGcited by 2,301

Not on the map. Why?

Cited by 92 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.

Key facts. Cited by 2,301 papers in the map (data through 7 Oct 2026). Builds on 43 of its 64 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

43 of 64 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
31 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  7. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  8. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  9. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  10. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  11. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  12. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  13. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  14. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  15. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  16. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  17. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  18. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
  19. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  20. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  21. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  22. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  23. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 58
  24. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  25. 2020Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstructioncited by 38
  26. 2017Learning Latent Representations for Speech Generation and Transformationcited by 27
  27. 2020Improved Speech Representations with Multi-Target Autoregressive Predictive Codingcited by 23
  28. 2019From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognitioncited by 20
  29. 2020A Convolutional Deep Markov Model for Unsupervised Speech Representation Learningcited by 18
  30. 2020Semi-Supervised Speech Recognition via Local Prior Matchingcited by 13
  31. 2021Contrastive Semi-supervised Learning for ASRcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 260; 2023: 430; 2024: 559; 2025: 580; 2026: 434.

2,301 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  5. 2022High Fidelity Neural Audio Compressioncited by 673
  6. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  12. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
38 more
  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  4. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  5. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  6. 2022AudioGen: Textually Guided Audio Generationcited by 292
  7. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  9. 2022Masked Autoencoders that Listencited by 271
  10. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  11. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  12. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  13. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  14. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  15. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  16. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  17. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  18. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  19. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  20. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  21. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  22. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  23. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  24. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  25. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  26. 2021Direct speech-to-speech translation with discrete unitscited by 148
  27. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  28. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  29. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  30. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  31. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  32. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  33. 2023End-to-End Speech Recognition: A Surveycited by 127
  34. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  35. 2022Generative Spoken Dialogue Language Modelingcited by 125
  36. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  37. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  38. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123

50 most cited of 2,301 All 2,301 (JSON)