mlsanityData to 14 Oct 2026

Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training

arXiv 2206.10125 ↗First posted 21 Jun 2022cs.CL cs.SD eess.AScited by 14

Not on the map. Why?

Abstract (arXiv)

Recently, masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. It usually requires a codebook obtained in an unsupervised way, making it less accurate and difficult to interpret. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance and also the pre-training efficiency, either through decoding with a hybrid ASR system to generate phoneme-level alignments (named PBERT), or performing clustering on the supervised speech features extracted from an end-to-end CTC model (named CTC clustering). Both the hybrid and CTC models are trained on the same small amount of labeled speech as used in fine-tuning. Experiments demonstrate significant superiority of our methods to various SSL and self-training baselines, with up to 17.0% relative WER reduction. Our pre-trained models also show good transferability in a non-ASR speech task.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

22 of 37 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  11. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  12. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
10 more
  1. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  2. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  3. 2019Self-Training for End-to-End Speech Recognitioncited by 129
  4. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  5. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 83
  6. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  7. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  8. 2018wav2letter++: The Fastest Open-source Speech Recognition Systemcited by 42
  9. 2019From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognitioncited by 20
  10. 2021On lattice-free boosted MMI training of HMM and CTC-based full-context ASR modelscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 5; 2024: 0; 2025: 2.

14 papers.

Sort:

  1. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  2. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  3. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  4. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  5. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  6. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  7. 2023Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representationcited by 7
  8. 2022token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Textcited by 3
  9. 2022Biased Self-supervised learning for ASRcited by 3
  10. 2023Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognitioncited by 3
  11. 2025VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretrainingcited by 3
  12. 2022TriNet: stabilizing self-supervised learning from complete or slow collapse on ASRcited by 1
2 more
  1. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  2. 2022Guided contrastive self-supervised pre-training for automatic speech recognitioncited by 0