MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language Modelling methods for learning Speech Representations
Not on the map. Why?
Abstract (arXiv)
In recent years, self-supervised pre-training methods have gained significant traction in learning high-level information from raw speech. Among these methods, HuBERT has demonstrated SOTA performance in automatic speech recognition (ASR). However, HuBERT's performance lags behind data2vec due to disparities in pre-training strategies. In this paper, we propose (i) a Swap method to address pre-training and inference mismatch observed in HuBERT and (ii) incorporates Multicluster masked prediction loss for more effective utilization of the models capacity. The resulting method is, MS-HuBERT, an end-to-end self-supervised pre-training method for learning robust speech representations. It beats vanilla HuBERT on the ASR Librispeech benchmark on average by a 5% margin when evaluated on different finetuning splits. Additionally, we demonstrate that the learned embeddings obtained during pre-training encode essential information for improving performance of content based tasks such as ASR.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 9 Jun 2024.
Builds on
23 of 26 references.
References · 23
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
11 more
- 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
- 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
- 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
- 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
- 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
- 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
- 2018Improved Speech Enhancement with the Wave-U-Netcited by 23
- 2023Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Predictioncited by 17
- 2023Exploration on HuBERT with Multiple Resolutionscited by 7
- 2023Single Channel Speech Enhancement Using U-Net Spiking Neural Networkscited by 5
- 2023Analysing the Masked predictive coding training criterion for pre-training a Speech Representation Modelcited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.