mlsanityData to 7 Oct 2026

Wav2CLIP: Learning Robust Audio Representations From CLIP

arXiv 2110.11499 ↗First posted 21 Oct 2021cs.SD cs.LG eess.AScited by 210

Not on the map. Why?

Abstract (arXiv)

We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ~10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

24 of 34 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  11. 2017Look, Listen and Learncited by 452
  12. 2019Clotho: An Audio Captioning Datasetcited by 354
12 more
  1. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  2. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  3. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  5. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  6. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  7. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  8. 2021Contrastive Learning of Musical Representationscited by 81
  9. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  10. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  11. 2021Multi-Task Self-Supervised Pre-Training for Music Classificationcited by 18
  12. 2020A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysiscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 26; 2023: 63; 2024: 56; 2025: 39; 2026: 24.

210 papers.

Sort:

By area:

  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  2. 2023MusicLM: Generating Music From Textcited by 442
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  4. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  5. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  6. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  7. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  8. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  9. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  10. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  11. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  12. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
38 more
  1. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  2. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  3. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  4. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  5. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  6. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  7. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
  8. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  9. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  10. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  11. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  12. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  13. 2022IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Textcited by 36
  14. 2023Connecting Multi-modal Contrastive Representationscited by 36
  15. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 36
  16. 2021Sound-Guided Semantic Image Manipulationcited by 35
  17. 2022Sound-Guided Semantic Video Generationcited by 35
  18. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  19. 2023PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactionscited by 34
  20. 2022LidarCLIP or: How I Learned to Talk to Point Cloudscited by 33
  21. 2022Contrastive Audio-Language Learning for Musiccited by 32
  22. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  23. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  24. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  25. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  26. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  27. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  28. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  29. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  30. 2023Discovering Sounding Objects by Audio Queries for Audio Visual Segmentationcited by 27
  31. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  32. 2022I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervisioncited by 26
  33. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  34. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  35. 2025MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audiocited by 24
  36. 2022CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videoscited by 23
  37. 2023DiffDance: Cascaded Human Motion Diffusion Model for Dance Generationcited by 23
  38. 2022Learning Visual Styles from Audio-Visual Associationscited by 22

50 most cited of 210 All 210 (JSON)