# Data Augmenting Contrastive Learning of Speech Representations in the Time Domain

arXiv: https://arxiv.org/abs/2007.00991 · HTML: https://mlsanity.com/paper/2007.00991/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 30 of 63)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [Best-of-N Jailbreaking](https://mlsanity.com/paper/2412.03556/) (2024)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)
- [Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering](https://mlsanity.com/paper/2109.03381/) (2021)
- [AugLy: Data Augmentations for Robustness](https://mlsanity.com/paper/2201.06494/) (2022)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [textless-lib: a Library for Textless Spoken Language Processing](https://mlsanity.com/paper/2202.07359/) (2022)
- [Learning Transformer-based World Models with Contrastive Predictive Coding](https://mlsanity.com/paper/2503.04416/) (2025)
- [Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling](https://mlsanity.com/paper/2209.15483/) (2022)
- [End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021](https://mlsanity.com/paper/2105.04512/) (2021)
- [Self-Supervised Learning for speech recognition with Intermediate layer supervision](https://mlsanity.com/paper/2112.08778/) (2021)
- [CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representations](https://mlsanity.com/paper/2210.02592/) (2022)
- [Supervised Contrastive Learning for Accented Speech Recognition](https://mlsanity.com/paper/2107.00921/) (2021)
- [From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension](https://mlsanity.com/paper/2112.04735/) (2021)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [Wav2Vec-Aug: Improved self-supervised training with limited data](https://mlsanity.com/paper/2206.13654/) (2022)
- [Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning](https://mlsanity.com/paper/2212.08378/) (2022)
- [Multi-view Contrastive Self-Supervised Learning of Accounting Data Representations for Downstream Audit Tasks](https://mlsanity.com/paper/2109.11201/) (2021)
- [Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning](https://mlsanity.com/paper/2204.05148/) (2022)
- [Improving generalizability of distilled self-supervised speech processing models under distorted settings](https://mlsanity.com/paper/2210.07978/) (2022)

## Abstract (arXiv)

Contrastive Predictive Coding (CPC), based on predicting future segments of speech based on past segments is emerging as a powerful algorithm for representation learning of speech signal. However, it still under-performs other methods on unsupervised evaluation benchmarks. Here, we introduce WavAugment, a time-domain data augmentation library and find that applying augmentation in the past is generally more efficient and yields better performances than other methods. We find that a combination of pitch modification, additive noise and reverberation substantially increase the performance of CPC (relative improvement of 18-22%), beating the reference Libri-light results with 600 times less data. Using an out-of-domain dataset, time-domain data augmentation can push CPC to be on par with the state of the art on the Zero Speech Benchmark 2017. We also show that time-domain data augmentation consistently improves downstream limited-supervision phoneme classification tasks by a factor of 12-15% relative.