# Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models

arXiv: https://arxiv.org/abs/2404.06309 · HTML: https://mlsanity.com/paper/2404.06309/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 7 of its 94 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Learning Generalized Zero-Shot Learners for Open-Domain Image Geolocalization](https://mlsanity.com/paper/2302.00275/) (2023)
- [Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements](https://mlsanity.com/paper/2012.03478/) (2020)

## Built on by (most cited first, 2 of 2)

- [Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learning](https://mlsanity.com/paper/2503.22197/) (2025)
- [Class Similarity-Based Multimodal Classification under Heterogeneous Category Sets](https://mlsanity.com/paper/2506.09745/) (2025)

## Abstract (arXiv)

Audio-visual zero-shot learning methods commonly build on features extracted from pre-trained models, e.g. video or audio classification models. However, existing benchmarks predate the popularization of large multi-modal models, such as CLIP and CLAP. In this work, we explore such large pre-trained models to obtain features, i.e. CLIP for visual features, and CLAP for audio features. Furthermore, the CLIP and CLAP text encoders provide class label embeddings which are combined to boost the performance of the system. We propose a simple yet effective model that only relies on feed-forward neural networks, exploiting the strong generalization capabilities of the new audio, visual and textual features. Our framework achieves state-of-the-art performance on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL with our new features. Code and data available at: https://github.com/dkurzend/ClipClap-GZSL.