mlsanityData to 14 Oct 2026

Zero-shot audio captioning with audio-language model guidance and audio context keywords

arXiv 2311.08396 ↗First posted 14 Nov 2023eess.AS cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Zero-shot audio captioning aims at automatically generating descriptive textual captions for audio content without prior training for this task. Different from speech recognition which translates audio content that contains spoken language into text, audio captioning is commonly concerned with ambient sounds, or sounds produced by a human performing an action. Inspired by zero-shot image captioning methods, we propose ZerAuCap, a novel framework for summarising such general audio signals in a text caption without requiring task-specific training. In particular, our framework exploits a pre-trained large language model (LLM) for generating the text which is guided by a pre-trained audio-language model to produce captions that describe the audio content. Additionally, we use audio context keywords that prompt the language model to generate text that is broadly relevant to sounds. Our proposed framework achieves state-of-the-art results in zero-shot audio captioning on the AudioCaps and Clotho datasets. Our code is available at https://github.com/ExplainableML/ZerAuCap.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

32 of 42 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2019Clotho: An Audio Captioning Datasetcited by 358
  9. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  10. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  11. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  12. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
20 more
  1. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  2. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 113
  3. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 90
  4. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  5. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  6. 2021Audio Captioning Transformercited by 55
  7. 2023AutoAD: Movie Description in Contextcited by 50
  8. 2021Audio Retrieval with Natural Language Queriescited by 49
  9. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 49
  10. 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
  11. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
  12. 2021Investigating Local and Global Information for Automated Audio Captioning with Transfer Learningcited by 26
  13. 2022Zero-Shot Video Captioning with Evolving Pseudo-Tokenscited by 25
  14. 2019Audio Caption: Listen and Tellcited by 20
  15. 2021CL4AC: A Contrastive Loss for Audio Captioningcited by 16
  16. 2022Exploring Train and Test-Time Augmentations for Audio-Language Learningcited by 11
  17. 2022Audio-text Retrieval in Contextcited by 10
  18. 2022Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignmentcited by 8
  19. 2023Zero-Shot Audio Captioning via Audibility Guidancecited by 3
  20. 2023Zero-shot Translation of Attention Patterns in VQA Models to Natural Languagecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 3; 2026: 1.

11 papers.

  1. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  2. 2024Taming Data and Transformers for Audio Generationcited by 28
  3. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioningcited by 11
  4. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  5. 2024AlignCap: Aligning Speech Emotion Captioning to Human Preferencescited by 5
  6. 2023Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary Empirical Studycited by 4
  7. 2025HuMoCon: Concept Discovery for Human Motion Understandingcited by 4
  8. 2025Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gapcited by 3
  9. 2024An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignmentcited by 0
  10. 2025Classifier-Guided Captioning Across Modalitiescited by 0
  11. 2026COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddingscited by 0