mlsanityData to 7 Oct 2026

Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models

arXiv 2404.06309 ↗First posted 9 Apr 2024cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Audio-visual zero-shot learning methods commonly build on features extracted from pre-trained models, e.g. video or audio classification models. However, existing benchmarks predate the popularization of large multi-modal models, such as CLIP and CLAP. In this work, we explore such large pre-trained models to obtain features, i.e. CLIP for visual features, and CLAP for audio features. Furthermore, the CLIP and CLAP text encoders provide class label embeddings which are combined to boost the performance of the system. We propose a simple yet effective model that only relies on feed-forward neural networks, exploiting the strong generalization capabilities of the new audio, visual and textual features. Our framework achieves state-of-the-art performance on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL with our new features. Code and data available at: https://github.com/dkurzend/ClipClap-GZSL.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 94 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

7 of 94 references.

References · 7

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  4. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  5. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  6. 2023Learning Generalized Zero-Shot Learners for Open-Domain Image Geolocalizationcited by 41
  7. 2020Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movementscited by 22

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learningcited by 0
  2. 2025Class Similarity-Based Multimodal Classification under Heterogeneous Category Setscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts
  2. 2024PALM: Few-Shot Prompt Learning for Audio Language Models
  3. 2024An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment
  4. 2023Prompting Language-Informed Distribution for Compositional Zero-Shot Learning
  5. 2023Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP
  6. 2022Prompting Large Pre-trained Vision-Language Models For Compositional Concept Learning
  7. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models
  8. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning