mlsanityData to 7 Oct 2026

A Brief Overview of Unsupervised Neural Speech Representation Learning

arXiv 2203.01829 ↗First posted 1 Mar 2022eess.AS cs.LG cs.SDcited by 9

Not on the map. Why?

Abstract (arXiv)

Unsupervised representation learning for speech processing has matured greatly in the last few years. Work in computer vision and natural language processing has paved the way, but speech data offers unique challenges. As a result, methods from other domains rarely translate directly. We review the development of unsupervised representation learning for speech over the last decade. We identify two primary model categories: self-supervised methods and probabilistic latent variable models. We describe the models and develop a comprehensive taxonomy. Finally, we discuss and compare models from the two categories.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 84 of its 133 references that are in the map. First posted on arXiv 1 Mar 2022.

Builds on

84 of 133 references.

References · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
38 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  7. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  10. 2015Generating Sentences from a Continuous Spacecited by 607
  11. 2015Listen, Attend and Spellcited by 542
  12. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  13. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  14. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  15. 2016Ladder Variational Autoencoderscited by 315
  16. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  17. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  18. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  19. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  20. 2021Unsupervised Speech Recognitioncited by 147
  21. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  22. 2016Sequential Neural Models with Stochastic Layerscited by 115
  23. 2020Unsupervised pretraining transfers well across languagescited by 100
  24. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  25. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  26. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  27. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  28. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  29. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  30. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  31. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  32. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  33. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  34. 2020not-MIWAE: Deep Generative Modelling with Missing not at Random Datacited by 61
  35. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 58
  36. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  37. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  38. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 1; 2024: 1; 2025: 1; 2026: 1.

9 papers.

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  3. 2022Generative Spoken Dialogue Language Modelingcited by 125
  4. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  5. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  6. 2023Bigger is not Always Better: The Effect of Context Size on Speech Pre-Trainingcited by 2
  7. 2024Analytic Study of Text-Free Speech Synthesis for Raw Audio using a Self-Supervised Learning Modelcited by 1
  8. 2025MixedG2P-T5: G2P-free Speech Synthesis for Mixed-script texts using Speech Self-Supervised Learning and Language Modelcited by 0
  9. 2026Self-Supervised Learning for Speaker Recognition: A study and reviewcited by 0