mlsanityData to 7 Oct 2026

CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representations

arXiv 2210.02592 ↗First posted 5 Oct 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

While Self-Supervised Learning has helped reap the benefit of the scale from the available unlabeled data, the learning paradigms are continuously being bettered. We present a new pre-training strategy named ccc-wav2vec 2.0, which uses clustering and an augmentation-based cross-contrastive loss as its self-supervised objective. Through the clustering module, we scale down the influence of those negative examples that are highly similar to the positive. The Cross-Contrastive loss is computed between the encoder output of the original sample and the quantizer output of its augmentation and vice-versa, bringing robustness to the pre-training strategy. ccc-wav2vec 2.0 achieves up to 15.6% and 12.7% relative WER improvement over the baseline wav2vec 2.0 on the test-clean and test-other sets, respectively, of LibriSpeech, without the use of any language model. The proposed method also achieves up to 14.9% relative WER improvement over the baseline wav2vec 2.0 when fine-tuned on Switchboard data. We make all our codes publicly available on GitHub.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

33 of 40 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2020Contrastive Learning with Hard Negative Samplescited by 619
21 more
  1. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  2. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  3. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  4. 2020Debiased Contrastive Learningcited by 354
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  6. 2020Contrastive Clusteringcited by 264
  7. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  8. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  9. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  10. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  11. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 43
  12. 2021Solving Inefficiency of Self-supervised Representation Learningcited by 41
  13. 2021MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognitioncited by 38
  14. 2020Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learningcited by 33
  15. 2021The History of Speech Recognition to the Year 2030cited by 10
  16. 2022Wav2Vec-Aug: Improved self-supervised training with limited datacited by 9
  17. 2021Data Augmentation Methods for End-to-end Speech Recognition on Distant-Talk Scenarioscited by 6
  18. 2022Text Transformations in Contrastive Self-Supervised Learning: A Reviewcited by 5
  19. 2022Negative Selection by Clustering for Contrastive Learning in Human Activity Recognitioncited by 5
  20. 2020Improving Noise Robustness of an End-to-End Neural Model for Automatic Speech Recognitioncited by 2
  21. 2021Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognitioncited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 5; 2025: 2.

11 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  3. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
  4. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  5. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
  6. 2022data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setupcited by 4
  7. 2024The Second DISPLACE Challenge : DIarization of SPeaker and LAnguage in Conversational Environmentscited by 4
  8. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  9. 2024Convexity-based Pruning of Speech Representation Modelscited by 2
  10. 2024Removing Speaker Information from Speech Representation using Variable-Length Soft Poolingcited by 1
  11. 2023SALTTS: Leveraging Self-Supervised Speech Representations for improved Text-to-Speech Synthesiscited by 0