mlsanityData to 7 Oct 2026

MelHuBERT: A simplified HuBERT on Mel spectrograms

arXiv 2211.09944 ↗First posted 17 Nov 2022cs.CL cs.LG cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

Self-supervised models have had great success in learning speech representations that can generalize to various downstream tasks. However, most self-supervised models require a large amount of compute and multiple GPUs to train, significantly hampering the development of self-supervised learning. In an attempt to reduce the computation of training, we revisit the training of HuBERT, a highly successful self-supervised model. We improve and simplify several key components, including the loss function, input representation, and training in multiple stages. Our model, MelHuBERT, is able to achieve favorable performance on phone recognition, speaker identification, and automatic speech recognition against HuBERT, while saving 31.2% of the pre-training time, or equivalently 33.5% MACs per one second speech. The code and pre-trained models are available in https://github.com/nervjack2/MelHuBERT.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

31 of 43 references.

References · 31

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2017Non-Autoregressive Neural Machine Translationcited by 457
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  12. 2022Masked Autoencoders that Listencited by 271
19 more
  1. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  2. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  4. 2018Speaker Recognition from Raw Waveform with SincNetcited by 166
  5. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  6. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  7. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  8. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  9. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  10. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  11. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  12. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  13. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 50
  14. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  15. 2019Improving Transformer-based Speech Recognition Using Unsupervised Pre-trainingcited by 29
  16. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  17. 2021Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learningcited by 10
  18. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  19. 2022On Compressing Sequences for Self-Supervised Speech Modelscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 4; 2025: 7.

15 papers.

Sort:

By area:

  1. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  2. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 34
  3. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  4. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 32
  5. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  6. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  7. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  8. 2025VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretrainingcited by 3
  9. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  10. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  11. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  12. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
3 more
  1. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  2. 2025Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformerscited by 1
  3. 2025Learning Speech Representations with Variational Predictive Codingcited by 0