mlsanityData to 14 Oct 2026

Learning Speech Representations with Variational Predictive Coding

arXiv 2601.00100 ↗First posted 31 Dec 2025eess.AS cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite being the best known objective for learning speech representations, the HuBERT objective has not been further developed and improved. We argue that it is the lack of an underlying principle that stalls the development, and, in this paper, we show that predictive coding under a variational view is the principle behind the HuBERT objective. Due to its generality, our formulation provides opportunities to improve parameterization and optimization, and we show two simple modifications that bring immediate improvements to the HuBERT objective. In addition, the predictive coding formulation has tight connections to various other objectives, such as APC, CPC, wav2vec, and BEST-RQ. Empirically, the improvement in pre-training brings significant improvements to four downstream tasks: phone classification, f0 tracking, speaker recognition, and automatic speech recognition, highlighting the importance of the predictive coding interpretation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

34 of 62 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  8. 2020On Layer Normalization in the Transformer Architecturecited by 922
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
22 more
  1. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  2. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  4. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  5. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  6. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  7. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  8. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  9. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  10. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 76
  11. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  12. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  13. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  14. 2018wav2letter++: The Fastest Open-source Speech Recognition Systemcited by 42
  15. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  16. 2019Improving Transformer-based Speech Recognition Using Unsupervised Pre-trainingcited by 30
  17. 2023Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Predictioncited by 17
  18. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  19. 2020Transformer based unsupervised pre-training for acoustic representation learningcited by 10
  20. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  21. 2018Information Theoretic Co-Trainingcited by 7
  22. 2024MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language Modelling methods for learning Speech Representationscited by 2

Built on by

Not cited yet.