Investigating the 'Autoencoder Behavior' in Speech Self-Supervised Models: a focus on HuBERT's Pretraining
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning has shown great success in Speech Recognition. However, it has been observed that finetuning all layers of the learned model leads to lower performance compared to resetting top layers. This phenomenon is attributed to the ''autoencoder'' behavior: top layers contain information closer to the input and are less suitable for tasks that require linguistic information, such as Speech Recognition.To better our understanding of this behavior, we propose to study the evolution of high-level information within the model during pretraining. We focus on the HuBERT model, which exhibits a less pronounced ''autoencoder'' behavior. By experimentally exploring various factors that may have an impact, we aim to improve the training procedure and enhance the top layers of HuBERT for high-level tasks.Furthermore, our experiments demonstrate that these improvements in the training procedure result in faster convergence and competitive performance on downstream tasks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 14 May 2024.
Builds on
20 of 23 references.
References · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
- 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
- 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
- 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
- 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
- 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
8 more
- 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
- 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
- 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
- 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19
- 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
- 2019Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognitioncited by 15
- 2022Speech Pre-training with Acoustic Piececited by 14
- 2022Are word boundaries useful for unsupervised language learning?cited by 8
Built on by
Citing papers per year: 2025: 3.
3 papers.