mlsanityData to 14 Oct 2026

RECAP: Retrieval-Augmented Audio Captioning

arXiv 2309.09836 ↗First posted 18 Sep 2023eess.AS cs.AI cs.CLcited by 27

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

We present RECAP (REtrieval-Augmented Audio CAPtioning), a novel and effective audio captioning system that generates captions conditioned on an input audio and other captions similar to the audio retrieved from a datastore. Additionally, our proposed method can transfer to any domain without the need for any additional fine-tuning. To generate a caption for an audio sample, we leverage an audio-text model CLAP to retrieve captions similar to it from a replaceable datastore, which are then used to construct a prompt. Next, we feed this prompt to a GPT-2 decoder and introduce cross-attention layers between the CLAP encoder and GPT-2 to condition the audio for caption generation. Experiments on two benchmark datasets, Clotho and AudioCaps, show that RECAP achieves competitive performance in in-domain settings and significant improvements in out-of-domain settings. Additionally, due to its capability to exploit a large text-captions-only datastore in a training-free fashion, RECAP shows unique capabilities of captioning novel audio events never seen during training and compositional audios with multiple events. To promote research in this space, we also release 150,000+ new weakly labeled captions for AudioSet, AudioCaps, and Clotho.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 16 of its 24 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

16 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Retrieval-augmented Image Captioning16 Feb
  2. 2022Retrieval-Augmented Transformer for Image Captioning26 Jul

Other references · 14

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  5. 2019Clotho: An Audio Captioning Datasetcited by 358
  6. 2022A Survey on Retrieval-Augmented Text Generationcited by 158
  7. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  8. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  9. 2021Audio Captioning Transformercited by 55
  10. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
  11. 2023Prefix tuning for automated audio captioningcited by 27
  12. 2021Investigating Local and Global Information for Automated Audio Captioning with Transfer Learningcited by 26
2 more
  1. 2021Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularizationcited by 12
  2. 2023A Whisper transformer for audio captioning trained with synthetic captions and transfer learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 12; 2026: 5.

27 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  2. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  3. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  4. 2024Taming Data and Transformers for Audio Generationcited by 28
  5. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognitioncited by 14
  6. 2024ReCLAP: Improving Zero Shot Audio Classification by Describing Soundscited by 12
  7. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioningcited by 11
  8. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  9. 2025Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesiscited by 4
  10. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processingcited by 4
  11. 2024Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learningcited by 3
  12. 2025CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizercited by 3
15 more
  1. 2025GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Modelscited by 3
  2. 2025Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gapcited by 3
  3. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  4. 2025MCA: Modality Composition Awareness for Robust Composed Multimodal Retrievalcited by 2
  5. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  6. 2024EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generationcited by 1
  7. 2025A General Retrieval-Augmented Generation Framework for Multimodal Case-Based Reasoning Applicationscited by 1
  8. 2025Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioningcited by 1
  9. 2025Classifier-Guided Captioning Across Modalitiescited by 0
  10. 2025Expressive Speech Retrieval using Natural Language Descriptions of Speaking Stylecited by 0
  11. 2025M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generationcited by 0
  12. 2026The Language of Touch: Translating Vibrations into Text with Dual-Branch Learningcited by 0
  13. 2026COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddingscited by 0
  14. 2026Mental Damage: Caption Poisoning Attacks on Retrieval-Augmented Text-to-Music Generationcited by 0
  15. 2026AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval
  2. 2024Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval
  3. 2023Retrieval-Augmented Text-to-Audio Generation
  4. 2026AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech
  5. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining
  6. 2024Audiobox TTA-RAG: Improving Zero-Shot and Few-Shot Text-To-Audio with Retrieval-Augmented Generation
  7. 2024Towards Diverse and Efficient Audio Captioning via Diffusion Models
  8. 2026AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward