MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models
Not on the map. Why?
Abstract (arXiv)
Automated Audio Captioning (AAC) generates captions for audio clips but faces challenges due to limited datasets compared to image captioning. To overcome this, we propose the zero-shot AAC system that leverages pre-trained models, eliminating the need for extensive training. Our approach uses a pre-trained audio CLIP model to extract auditory features and generate a structured prompt, which guides a Large Language Model (LLM) in caption generation. Unlike traditional greedy decoding, our method refines token selection through the audio CLIP model, ensuring alignment with the audio content. Experimental results demonstrate a 35% improvement in NLG mean score (from 4.7 to 7.3) using MAGIC search with the WavCaps model. The performance is heavily influenced by the audio-text matching model and keyword selection, with optimal results achieved using a single keyword prompt, and a 50% performance drop when no keyword list is used.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 16 Sep 2025.
Builds on
26 of 39 references.
References · 26
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
- 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
- 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
- 2019Clotho: An Audio Captioning Datasetcited by 358
- 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
- 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
- 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
- 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
- 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
14 more
- 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 90
- 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
- 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
- 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
- 2020A Transformer-based Audio Captioning Model with Keyword Estimationcited by 24
- 2022Automated Audio Captioning: An Overview of Recent Progress and New Challengescited by 23
- 2022Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learningcited by 15
- 2024Measuring the Accuracy of Automatic Speech Recognition Solutionscited by 15
- 2022Beyond the Status Quo: A Contemporary Survey of Advances and Challenges in Audio Captioningcited by 10
- 2020Listen carefully and tell: an audio captioning system based on residual learning and gammatone audio representationcited by 4
- 2023DiffSED: Sound Event Detection with Denoising Diffusioncited by 4
- 2023ATGNN: Audio Tagging Graph Neural Networkcited by 4
- 2024Improving Acoustic Scene Classification in Low-Resource Conditionscited by 2
- 2024VI-PANN: Harnessing Transfer Learning and Uncertainty-Aware Variational Inference for Improved Generalization in Audio Pattern Recognitioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.