mlsanityData to 7 Oct 2026

Generative Pre-Training for Speech with Autoregressive Predictive Coding

arXiv 1910.12607 ↗First posted 23 Oct 2019eess.AS cs.CL cs.LGcited by 82

Not on the map. Why?

Abstract (arXiv)

Learning meaningful and general representations from unannotated speech that are applicable to a wide range of tasks remains challenging. In this paper we propose to use autoregressive predictive coding (APC), a recently proposed self-supervised objective, as a generative pre-training approach for learning meaningful, non-specific, and transferable speech representations. We pre-train APC on large-scale unlabeled data and conduct transfer learning experiments on three speech applications that require different information about speech characteristics to perform well: speech recognition, speech translation, and speaker identification. Extensive experiments show that APC not only outperforms surface features (e.g., log Mel spectrograms) and other popular representation learning methods on all three tasks, but is also effective at reducing downstream labeled data size and model parameters. We also investigate the use of Transformers for modeling APC and find it superior to RNNs.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 27; 2023: 15; 2024: 7; 2025: 4; 2026: 1.

82 papers.

Sort:

By area:

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  4. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  7. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  8. 2022Generative Spoken Dialogue Language Modelingcited by 125
  9. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  10. 2022Audio Self-supervised Learning: A Surveycited by 65
  11. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 63
  12. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
38 more
  1. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  2. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 43
  3. 2021Momentum Pseudo-Labeling for Semi-Supervised Speech Recognitioncited by 37
  4. 2022Self-supervised models of audio effectively explain human cortical responses to speechcited by 37
  5. 2021Live Speech Portraits: Real-Time Photorealistic Talking-Head Animationcited by 36
  6. 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 35
  7. 2021Contrastively Disentangled Sequential Variational Autoencodercited by 34
  8. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  9. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  10. 2023RepCodec: A Speech Representation Codec for Speech Tokenizationcited by 27
  11. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  12. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  13. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 25
  14. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  15. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  16. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  17. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  18. 2021Scaling Laws for Acoustic Modelscited by 20
  19. 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
  20. 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19
  21. 2021Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learningcited by 16
  22. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  23. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 15
  24. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  25. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  26. 2023ADCNet: Learning from Raw Radar Data via Distillationcited by 14
  27. 2023ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMscited by 13
  28. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  29. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  30. 2021Contrastive Semi-supervised Learning for ASRcited by 11
  31. 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASRcited by 11
  32. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11
  33. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
  34. 2022A Comparative Study of Self-supervised Speech Representation Based Voice Conversioncited by 9
  35. 2022Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognitioncited by 8
  36. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  37. 2021DECAR: Deep Clustering for learning general-purpose Audio Representationscited by 7
  38. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7

50 most cited of 82 All 82 (JSON)