mlsanityData to 7 Oct 2026

Self-supervised Learning with Random-projection Quantizer for Speech Recognition

arXiv 2202.01855 ↗First posted 3 Feb 2022cs.CL cs.SD eess.AScited by 158

Not on the map. Why?

Abstract (arXiv)

We present a simple and effective self-supervised learning approach for speech recognition. The approach learns a model to predict the masked speech signals, in the form of discrete labels generated with a random-projection quantizer. In particular the quantizer projects speech inputs with a randomly initialized matrix, and does a nearest-neighbor lookup in a randomly-initialized codebook. Neither the matrix nor the codebook is updated during self-supervised learning. Since the random-projection quantizer is not trained and is separated from the speech recognition model, the design makes the approach flexible and is compatible with universal speech recognition architecture. On LibriSpeech our approach achieves similar word-error-rates as previous work using self-supervised learning with non-streaming models, and provides lower word-error-rates and latency than wav2vec 2.0 and w2v-BERT with streaming models. On multilingual tasks the approach also provides significant improvement over wav2vec 2.0 and w2v-BERT.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 3 Feb 2022.

Builds on

23 of 31 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
11 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  2. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  3. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  4. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  5. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  6. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  7. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  8. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  9. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 66
  10. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  11. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 36; 2024: 45; 2025: 41; 2026: 20.

158 papers.

Sort:

By area:

  1. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  3. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  4. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  5. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  6. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  7. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  8. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  9. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  10. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  11. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  12. 2024Restructuring Vector Quantization with the Rotation Trickcited by 49
38 more
  1. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  3. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
  4. 2024Foundation Models for Music: A Surveycited by 34
  5. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  6. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  7. 2022Pseudo Label Is Better Than Human Labelcited by 25
  8. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  9. 2023Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representationscited by 22
  10. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  11. 2024Retrieval Augmented End-to-End Spoken Dialog Modelscited by 20
  12. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  13. 2025Fun-ASR Technical Reportcited by 19
  14. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  15. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  16. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  17. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  18. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Modelscited by 16
  19. 2022Light-weight probing of unsupervised representations for Reinforcement Learningcited by 15
  20. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 15
  21. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  22. 2024HeAR -- Health Acoustic Representationscited by 15
  23. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  24. 2024Anatomy of Industrial Scale Multilingual ASRcited by 14
  25. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  26. 2023Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervisioncited by 13
  27. 2025Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generationcited by 13
  28. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  29. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  30. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  31. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  32. 2022Towards Improved Zero-shot Voice Conversion with Conditional DSVAEcited by 10
  33. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASRcited by 10
  34. 2023Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASRcited by 10
  35. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  36. 2023Conformers are All You Need for Visual Speech Recognitioncited by 9
  37. 2024Open Implementation and Study of BEST-RQ for Speech Processingcited by 9
  38. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8

50 most cited of 158 All 158 (JSON)