mlsanityData to 7 Oct 2026

Multi-Format Contrastive Learning of Audio Representations

arXiv 2103.06508 ↗First posted 11 Mar 2021cs.SD cs.LG eess.AScited by 39

Not on the map. Why?

Abstract (arXiv)

Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

26 of 34 references.

References · 26

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  8. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  9. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  10. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  11. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
14 more
  1. 2017Look, Listen and Learncited by 452
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  4. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  5. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  6. 2017Objects that Soundcited by 254
  7. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  8. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  9. 2020Unsupervised pretraining transfers well across languagescited by 100
  10. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  11. 2017Unsupervised Learning of Semantic Audio Representationscited by 44
  12. 2020Learning Robust and Multilingual Speech Representationscited by 39
  13. 2018A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labelingcited by 37
  14. 2019Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervisioncited by 27

Built on by

Cited by year

Citing papers per year: 2021: 9; 2022: 14; 2023: 8; 2024: 6; 2025: 2.

39 papers.

Sort:

  1. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  2. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  3. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  4. 2022Audio Self-supervised Learning: A Surveycited by 65
  5. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  6. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
  7. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  8. 2021OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generationcited by 20
  9. 2022Sound Localization by Self-Supervised Time Delay Estimationcited by 18
  10. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  11. 2021Self-Supervised Learning of Audio Representations from Permutations with Differentiable Rankingcited by 14
  12. 2023CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perceptioncited by 14
27 more
  1. 2021Contrastive Neural Processes for Self-Supervised Learningcited by 9
  2. 2022SampleMatch: Drum Sample Retrieval by Musical Contextcited by 8
  3. 2022Better May Not Be Fairer: A Study on Subgroup Discrepancy in Image Classificationcited by 7
  4. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  5. 2021Self-Supervised Learning from Automatically Separated Sound Scenescited by 6
  6. 2024LC-Protonets: Multi-Label Few-Shot Learning for World Music Audio Taggingcited by 6
  7. 2024On the Effect of Data-Augmentation on Local Embedding Properties in the Contrastive Learning of Music Audio Representationscited by 4
  8. 2023XAI-based Comparison of Input Representations for Audio Event Classificationcited by 3
  9. 2023Pretraining Representations for Bioacoustic Few-shot Detection using Supervised Contrastive Learningcited by 2
  10. 2023Exploring Self-Supervised Contrastive Learning of Spatial Sound Event Representationcited by 2
  11. 2023Disentangling the Effects of Data Augmentation and Format Transform in Self-Supervised Learning of Image Representationscited by 2
  12. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  13. 2024Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Musiccited by 2
  14. 2021Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....cited by 1
  15. 2022A Study on Robustness to Perturbations for Representations of Environmental Soundcited by 1
  16. 2024Contrastive Learning from Synthetic Audio Doppelg"angerscited by 1
  17. 2021The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learningcited by 0
  18. 2021Learning source-aware representations of music in a discrete latent spacecited by 0
  19. 2022Self-Supervised Beat Tracking in Musical Signals with Polyphonic Contrastive Learningcited by 0
  20. 2022Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 0
  21. 2022Learning Audio Representations with MLPscited by 0
  22. 2022End-to-End Audio Strikes Back: Boosting Augmentations Towards An Efficient Audio Classification Networkcited by 0
  23. 2022One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hallcited by 0
  24. 2023Content Adaptive Front End For Audio Classificationcited by 0
  25. 2023Diverse Audio Embeddings -- Bringing Features Back Outperforms CLAP!cited by 0
  26. 2025SS-DPPN: A self-supervised dual-path foundation model for the generalizable cardiac audio representationcited by 0
  27. 2025Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projectionscited by 0