# Autoregressive Co-Training for Learning Discrete Speech Representations

arXiv: https://arxiv.org/abs/2203.15840 · HTML: https://mlsanity.com/paper/2203.15840/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019)
- [DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization](https://mlsanity.com/paper/2012.06659/) (2020)
- [Vector-Quantized Autoregressive Predictive Coding](https://mlsanity.com/paper/2005.08392/) (2020)
- [Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge](https://mlsanity.com/paper/2005.09409/) (2020)
- [Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies](https://mlsanity.com/paper/2011.00406/) (2020)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [Aligned Contrastive Predictive Coding](https://mlsanity.com/paper/2104.11946/) (2021)
- [Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis](https://mlsanity.com/paper/1807.11470/) (2018)
- [Discrete Latent Variable Representations for Low-Resource Text Classification](https://mlsanity.com/paper/2006.06226/) (2020)
- [A Comparison of Discrete Latent Variable Models for Speech Representation Learning](https://mlsanity.com/paper/2010.14230/) (2020)
- [Information Theoretic Co-Training](https://mlsanity.com/paper/1802.07572/) (2018)
- [Building DNN Acoustic Models for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1406.7806/) (2014)

## Built on by (most cited first, 8 of 8)

- [DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning](https://mlsanity.com/paper/2305.10005/) (2023)
- [Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering](https://mlsanity.com/paper/2305.11072/) (2023)
- [MelHuBERT: A simplified HuBERT on Mel spectrograms](https://mlsanity.com/paper/2211.09944/) (2022)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective](https://mlsanity.com/paper/2401.08833/) (2024)
- [Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models](https://mlsanity.com/paper/2210.16659/) (2022)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [Learning Speech Representations with Variational Predictive Coding](https://mlsanity.com/paper/2601.00100/) (2025)

## Abstract (arXiv)

While several self-supervised approaches for learning discrete speech representation have been proposed, it is unclear how these seemingly similar approaches relate to each other. In this paper, we consider a generative model with discrete latent variables that learns a discrete representation for speech. The objective of learning the generative model is formulated as information-theoretic co-training. Besides the wide generality, the objective can be optimized with several approaches, subsuming HuBERT-like training and vector quantization for learning discrete representation. Empirically, we find that the proposed approach learns discrete representation that is highly correlated with phonetic units, more correlated than HuBERT-like training and vector quantization.