mlsanityData to 7 Oct 2026

MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets

arXiv 2211.07321 ↗First posted 14 Nov 2022cs.CL cs.AI cs.SDcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide a new perspective on self-supervised speech models from how the training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extractor (On-TE). Based on this, we propose a new multi-tasking learning framework for self-supervised learning, MT4SSL, which stands for Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets. MT4SSL uses the K-means algorithm as an Off-TE and a teacher network without gradients as an On-TE, respectively. Our model outperforms previous SSL methods by nontrivial margins on the LibriSpeech benchmark, and is comparable to or even better than the best-performing models with fewer data. Furthermore, we find that using both Off-TE and On-TE results in better convergence in the pre-training phase. With both effectiveness and efficiency, we think doing multi-task learning on self-supervised speech models from our perspective is a promising trend.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

28 of 33 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
16 more
  1. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  2. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  3. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  4. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  5. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  6. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  7. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  8. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  9. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  10. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  11. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  12. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  13. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  14. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  15. 2022Speech Pre-training with Acoustic Piececited by 14
  16. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 3.

8 papers.

  1. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  2. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  3. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  4. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 32
  5. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  6. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  7. 2023Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representationcited by 7
  8. 2023Front-End Adapter: Adapting Front-End Input of Speech based Self-Supervised Learning for Speech Recognitioncited by 0