mlsanityData to 7 Oct 2026

AudioCLIP: Extending CLIP to Image, Text and Audio

arXiv 2106.13043 ↗First posted 24 Jun 2021cs.SD cs.CV eess.AScited by 335

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models. In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion. AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively). Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.

Key facts. Cited by 335 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

19 of 34 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2021AST: Audio Spectrogram Transformercited by 634
  7. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  8. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  9. 2016Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classificationcited by 84
  10. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  11. 2017Learning from Between-class Examples for Deep Sound Recognitioncited by 53
  12. 2020Rethinking CNN Models for Audio Classificationcited by 48
7 more
  1. 2020ESResNet: Environmental Sound Classification Based on Visual Domain Modelscited by 31
  2. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 31
  3. 2021ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognitioncited by 29
  4. 2021ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audiocited by 24
  5. 2020Zero-Shot Audio Classification via Semantic Embeddingscited by 21
  6. 2020A Sequential Self Teaching Approach for Improving Generalization in Sound Event Recognitioncited by 12
  7. 2019Zero-Shot Audio Classification Based on Class Label Embeddingscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 29; 2023: 83; 2024: 88; 2025: 80; 2026: 51.

335 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  3. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  4. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  5. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  7. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  8. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  9. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  10. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  11. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  12. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
38 more
  1. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  2. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  5. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  6. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  7. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  9. 2023Joint Audio and Speech Understandingcited by 99
  10. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  11. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  12. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  13. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  14. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  15. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  16. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  17. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  18. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  19. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  20. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  21. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  22. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  23. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  24. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  25. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  26. 2023CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Modelcited by 47
  27. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 46
  28. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 45
  29. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  30. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  31. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  32. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 38
  33. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  34. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  35. 2023VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Biascited by 37
  36. 2023Connecting Multi-modal Contrastive Representationscited by 36
  37. 2021Sound-Guided Semantic Image Manipulationcited by 35
  38. 2022Sound-Guided Semantic Video Generationcited by 35

50 most cited of 335 All 335 (JSON)