# RECAP: Retrieval-Augmented Audio Captioning

arXiv: https://arxiv.org/abs/2309.09836 · HTML: https://mlsanity.com/paper/2309.09836/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 16 of 16)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation](https://mlsanity.com/paper/2209.15323/) (2022)
- [Audio Retrieval with Natural Language Queries: A Benchmark Study](https://mlsanity.com/paper/2112.09418/) (2021)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Retrieval-Augmented Transformer for Image Captioning](https://mlsanity.com/paper/2207.13162/) (2022)
- [Retrieval-augmented Image Captioning](https://mlsanity.com/paper/2302.08268/) (2023)
- [Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval](https://mlsanity.com/paper/2012.07331/) (2020)
- [Prefix tuning for automated audio captioning](https://mlsanity.com/paper/2303.17489/) (2023)
- [Investigating Local and Global Information for Automated Audio Captioning with Transfer Learning](https://mlsanity.com/paper/2102.11457/) (2021)
- [Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization](https://mlsanity.com/paper/2108.04692/) (2021)
- [A Whisper transformer for audio captioning trained with synthetic captions and transfer learning](https://mlsanity.com/paper/2305.09690/) (2023)

## Built on by (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [Speech Retrieval-Augmented Generation without Automatic Speech Recognition](https://mlsanity.com/paper/2412.16500/) (2024)
- [ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds](https://mlsanity.com/paper/2409.09213/) (2024)
- [DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning](https://mlsanity.com/paper/2410.09472/) (2024)
- [Audio Dialogues: Dialogues dataset for audio and music understanding](https://mlsanity.com/paper/2404.07616/) (2024)
- [Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2501.06467/) (2025)
- [REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing](https://mlsanity.com/paper/2506.16444/) (2025)
- [Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning](https://mlsanity.com/paper/2410.10913/) (2024)
- [CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer](https://mlsanity.com/paper/2506.00800/) (2025)
- [GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models](https://mlsanity.com/paper/2506.17886/) (2025)
- [Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gap](https://mlsanity.com/paper/2510.11330/) (2025)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval](https://mlsanity.com/paper/2510.15543/) (2025)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation](https://mlsanity.com/paper/2410.12028/) (2024)
- [A General Retrieval-Augmented Generation Framework for Multimodal Case-Based Reasoning Applications](https://mlsanity.com/paper/2501.05030/) (2025)
- [Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning](https://mlsanity.com/paper/2502.05435/) (2025)
- [Classifier-Guided Captioning Across Modalities](https://mlsanity.com/paper/2501.03183/) (2025)
- [Expressive Speech Retrieval using Natural Language Descriptions of Speaking Style](https://mlsanity.com/paper/2508.11187/) (2025)
- [M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation](https://mlsanity.com/paper/2512.20136/) (2025)
- [The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning](https://mlsanity.com/paper/2603.26804/) (2026)
- [COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings](https://mlsanity.com/paper/2605.29628/) (2026)
- [Mental Damage: Caption Poisoning Attacks on Retrieval-Augmented Text-to-Music Generation](https://mlsanity.com/paper/2605.30365/) (2026)
- [AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models](https://mlsanity.com/paper/2606.14591/) (2026)

## Abstract (arXiv)

We present RECAP (REtrieval-Augmented Audio CAPtioning), a novel and effective audio captioning system that generates captions conditioned on an input audio and other captions similar to the audio retrieved from a datastore. Additionally, our proposed method can transfer to any domain without the need for any additional fine-tuning. To generate a caption for an audio sample, we leverage an audio-text model CLAP to retrieve captions similar to it from a replaceable datastore, which are then used to construct a prompt. Next, we feed this prompt to a GPT-2 decoder and introduce cross-attention layers between the CLAP encoder and GPT-2 to condition the audio for caption generation. Experiments on two benchmark datasets, Clotho and AudioCaps, show that RECAP achieves competitive performance in in-domain settings and significant improvements in out-of-domain settings. Additionally, due to its capability to exploit a large text-captions-only datastore in a training-free fashion, RECAP shows unique capabilities of captioning novel audio events never seen during training and compositional audios with multiple events. To promote research in this space, we also release 150,000+ new weakly labeled captions for AudioSet, AudioCaps, and Clotho.