mlsanityData to 7 Oct 2026

Aligned Contrastive Predictive Coding

arXiv 2104.11946 ↗First posted 24 Apr 2021cs.LG cs.SD eess.AScited by 15

Not on the map. Why?

Abstract (arXiv)

We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned. In this way, the prediction network solves a simpler task of predicting the next symbols, but not their exact timing, while the encoding network is trained to produce piece-wise constant latent codes. We evaluate the model on a speech coding task and demonstrate that the proposed Aligned Contrastive Predictive Coding (ACPC) leads to higher linear phone prediction accuracy and lower ABX error rates, while being slightly faster to train due to the reduced number of prediction heads.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 13 of its 22 references that are in the map. First posted on arXiv 24 Apr 2021.

Builds on

13 of 22 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2015Attention-Based Models for Speech Recognitioncited by 400
  6. 2020Unsupervised pretraining transfers well across languagescited by 100
  7. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  8. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  9. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  10. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  11. 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24
  12. 2021Variable-rate discrete representation learningcited by 24
1 more
  1. 2020A Convolutional Deep Markov Model for Unsupervised Speech Representation Learningcited by 18

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 9; 2023: 3; 2024: 1.

15 papers.

Sort:

  1. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  2. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  3. 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
  4. 2021Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and wordscited by 18
  5. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
  6. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  7. 2022On Compressing Sequences for Self-Supervised Speech Modelscited by 7
  8. 2022Similarity and Content-based Phonetic Self Attention for Speech Recognitioncited by 3
  9. 2022Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognitioncited by 2
  10. 2022Distribution-based Emotion Recognition in Conversationcited by 2
  11. 2023Audio-Visual Neural Syntax Acquisitioncited by 2
  12. 2022Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Modelscited by 1
3 more
  1. 2023Towards Matching Phones and Speech Representationscited by 1
  2. 2024A Simple HMM with Self-Supervised Representations for Phone Segmentationcited by 1
  3. 2023Self-supervised adversarial masking for 3D point cloud representation learningcited by 0