# Improved Speech Representations with Multi-Target Autoregressive Predictive Coding

arXiv: https://arxiv.org/abs/2004.05274 · HTML: https://mlsanity.com/paper/2004.05274/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2020.

## Built on by (most cited first, 23 of 23)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Unified Speech-Text Pre-training for Speech Translation and Recognition](https://mlsanity.com/paper/2204.05409/) (2022)
- [LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech](https://mlsanity.com/paper/2104.11462/) (2021)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2210.08634/) (2022)
- [ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer](https://mlsanity.com/paper/2305.12708/) (2023)
- [Don't speak too fast: The impact of data bias on self-supervised speech models](https://mlsanity.com/paper/2110.07957/) (2021)
- [On the Robustness of Arabic Speech Dialect Identification](https://mlsanity.com/paper/2306.03789/) (2023)
- [End2End Acoustic to Semantic Transduction](https://mlsanity.com/paper/2102.01013/) (2021)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)
- [Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2211.06173/) (2022)
- [Federated Self-Supervised Learning for Acoustic Event Classification](https://mlsanity.com/paper/2203.11997/) (2022)
- [Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction](https://mlsanity.com/paper/2507.09834/) (2025)
- [Fast Development of ASR in African Languages using Self Supervised Speech Representation Learning](https://mlsanity.com/paper/2103.08993/) (2021)
- [General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Framework](https://mlsanity.com/paper/2102.01930/) (2021)
- [Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language augmentation for Low Resource Self-Supervised Speech Models](https://mlsanity.com/paper/2309.12763/) (2023)

## Abstract (arXiv)

Training objectives based on predictive coding have recently been shown to be very effective at learning meaningful representations from unlabeled speech. One example is Autoregressive Predictive Coding (Chung et al., 2019), which trains an autoregressive RNN to generate an unseen future frame given a context such as recent past frames. The basic hypothesis of these approaches is that hidden states that can accurately predict future frames are a useful representation for many downstream tasks. In this paper we extend this hypothesis and aim to enrich the information encoded in the hidden states by training the model to make more accurate future predictions. We propose an auxiliary objective that serves as a regularization to improve generalization of the future frame prediction task. Experimental results on phonetic classification, speech recognition, and speech translation not only support the hypothesis, but also demonstrate the effectiveness of our approach in learning representations that contain richer phonetic content.