mlsanityData to 7 Oct 2026

Contrastive Learning of General-Purpose Audio Representations

arXiv 2010.10915 ↗First posted 21 Oct 2020cs.SD cs.LG eess.AScited by 168

Not on the map. Why?

Abstract (arXiv)

We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. Our approach is based on contrastive learning: it learns a representation which assigns high similarity to audio segments extracted from the same recording while assigning lower similarity to segments from different recordings. We build on top of recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement self-supervised model of audio. We pre-train embeddings on the large-scale Audioset database and transfer these representations to 9 diverse classification tasks, including speech, music, animal sounds, and acoustic scenes. We show that despite its simplicity, our method significantly outperforms previous self-supervised systems. We furthermore conduct ablation studies to identify key design choices and release a library to pre-train and fine-tune COLA models.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 44; 2023: 30; 2024: 37; 2025: 22; 2026: 9.

168 papers.

Sort:

By area:

  1. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  2. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  3. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  4. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  5. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  6. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  7. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  8. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  9. 2021Contrastive Learning of Musical Representationscited by 81
  10. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  11. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  12. 2022Audio Self-supervised Learning: A Surveycited by 65
38 more
  1. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  2. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  3. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  4. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  5. 2022Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentationcited by 37
  6. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
  7. 2021Universal Paralinguistic Speech Representations Using Self-Supervised Conformerscited by 32
  8. 2022Contrastive Audio-Language Learning for Musiccited by 32
  9. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  10. 2021Max-Margin Contrastive Learningcited by 27
  11. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  12. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  13. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  14. 2025Perch 2.0: The Bittern Lesson for Bioacousticscited by 25
  15. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  16. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 23
  17. 2024Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarkingcited by 23
  18. 2022COCOA: Cross Modality Contrastive Learning for Sensor Datacited by 21
  19. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representationscited by 19
  20. 2023Real-Time Idling Vehicles Detection using Combined Audio-Visual Deep Learningcited by 17
  21. 2021Exploring Self-Supervised Representation Ensembles for COVID-19 Cough Classificationcited by 16
  22. 2021Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?cited by 16
  23. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
  24. 2022Pretraining Respiratory Sound Representations using Metadata and Contrastive Learningcited by 15
  25. 2023CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perceptioncited by 14
  26. 2024COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representationscited by 14
  27. 2024animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacousticscited by 14
  28. 2022ATST: Audio Representation Learning with Teacher-Student Transformercited by 13
  29. 2022Labeling Chaos to Learning Harmony: Federated Learning with Noisy Labelscited by 13
  30. 2022ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classificationcited by 13
  31. 2021One Billion Audio Sounds from GPU-enabled Modular Synthesiscited by 11
  32. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11
  33. 2023Self-Supervised Learning for Few-Shot Bird Sound Classificationcited by 11
  34. 2021Multi-Modal Pre-Training for Automated Speech Recognitioncited by 10
  35. 2022Contrastive Learning with Positive-Negative Frame Mask for Music Representationcited by 10
  36. 2021Teaching keyword spotters to spot new keywords with limited examplescited by 9
  37. 2022BYOL-S: Learning Self-supervised Speech Representations by Bootstrappingcited by 9
  38. 2023Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers?cited by 9

50 most cited of 168 All 168 (JSON)