mlsanityData to 14 Oct 2026

Self-Supervised Learning for speech recognition with Intermediate layer supervision

arXiv 2112.08778 ↗First posted 16 Dec 2021eess.AS cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Recently, pioneer work finds that speech pre-trained models can solve full-stack speech processing tasks, because the model utilizes bottom layers to learn speaker-related information and top layers to encode content-related information. Since the network capacity is limited, we believe the speech recognition performance could be further improved if the model is dedicated to audio content information learning. To this end, we propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL), which forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly, which achieves a 23.5%/11.6% relative word error rate reduction in the w/o language model setting for base/large models. Detailed analysis shows the bottom layers of our model have a better correlation with phonetic units, which is consistent with our intuition and explains the success of our method for ASR.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

28 of 31 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
16 more
  1. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  2. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  3. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  4. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  5. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  6. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  7. 2021Unsupervised Speech Recognitioncited by 148
  8. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 130
  9. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  10. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  11. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  12. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  13. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  14. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  15. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  16. 2020Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstructioncited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 5; 2024: 1; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  3. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  4. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  5. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  6. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  7. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
  8. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  9. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  10. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  11. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  12. 2023Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representationscited by 4
3 more
  1. 2024MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyondcited by 4
  2. 2023EM-Network: Oracle Guided Self-distillation for Sequence Learningcited by 1
  3. 2026MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Ratescited by 0