mlsanityData to 7 Oct 2026

Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning

arXiv 2410.10913 ↗First posted 14 Oct 2024cs.SD cs.AI eess.AScited by 3

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-augmented generation can improve audio captioning by incorporating relevant audio-text pairs from a knowledge base. Existing methods typically rely solely on the input audio as a unimodal retrieval query. In contrast, we propose Generation-Assisted Multimodal Querying, which generates a text description of the input audio to enable multimodal querying. This approach aligns the query modality with the audio-text structure of the knowledge base, leading to more effective retrieval. Furthermore, we introduce a novel progressive learning strategy that gradually increases the number of interleaved audio-text pairs to enhance the training process. Our experiments on AudioCaps, Clotho, and Auto-ACD demonstrate that our approach achieves state-of-the-art results across these benchmarks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 30 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

27 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning12 Oct
  2. 2023RECAP: Retrieval-Augmented Audio Captioning18 Sep
  3. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  6. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  7. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  8. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  9. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  10. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  11. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  12. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
12 more
  1. 2020How Context Affects Language Models' Factual Predictionscited by 181
  2. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  3. 2021Audio Retrieval with Natural Language Queriescited by 49
  4. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  5. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  6. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  7. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  8. 2024Taming Data and Transformers for Audio Generationcited by 28
  9. 2023Prefix tuning for automated audio captioningcited by 26
  10. 2022Noise-aware Learning from Web-crawled Image-Text Data for Image Captioningcited by 22
  11. 2022Exploring Train and Test-Time Augmentations for Audio-Language Learningcited by 11
  12. 2024Revisiting Deep Audio-Text Retrieval Through the Lens of Transportationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergencecited by 1
  2. 2026Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Losscited by 0
  3. 2026Mental Damage: Caption Poisoning Attacks on Retrieval-Augmented Text-to-Music Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Retrieval-Augmented Text-to-Audio Generation
  2. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
  3. 2026SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval
  4. 2026Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval
  5. 2024Audiobox TTA-RAG: Improving Zero-Shot and Few-Shot Text-To-Audio with Retrieval-Augmented Generation
  6. 2025M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
  7. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models
  8. 2024Towards Diverse and Efficient Audio Captioning via Diffusion Models