mlsanityData to 14 Oct 2026

WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models

arXiv 2502.14727 ↗First posted 20 Feb 2025cs.SD cs.AI eess.AScited by 18

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Retrieval Augmented Generation (RAG) has gained widespread adoption owing to its capacity to empower large language models (LLMs) to integrate external knowledge. However, existing RAG frameworks are primarily designed for text-based LLMs and rely on Automatic Speech Recognition to process speech input, which discards crucial audio information, risks transcription errors, and increases computational overhead. Therefore, we introduce WavRAG, the first retrieval augmented generation framework with native, end-to-end audio support. WavRAG offers two key features: 1) Bypassing ASR, WavRAG directly processes raw audio for both embedding and retrieval. 2) WavRAG integrates audio and text into a unified knowledge representation. Specifically, we propose the WavRetriever to facilitate the retrieval from a text-audio hybrid knowledge base, and further enhance the in-context capabilities of spoken dialogue models through the integration of chain-of-thought reasoning. In comparison to state-of-the-art ASR-Text RAG pipelines, WavRAG achieves comparable retrieval performance while delivering a 10x acceleration. Furthermore, WavRAG's unique text-audio hybrid retrieval capability extends the boundaries of RAG to the audio modality.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 30 of its 45 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

30 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognition21 Dec

Other references · 29

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  10. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  11. 2019ELI5: Long Form Question Answeringcited by 575
  12. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
17 more
  1. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  2. 2019Clotho: An Audio Captioning Datasetcited by 358
  3. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  4. 2024Graph Retrieval-Augmented Generation: A Surveycited by 324
  5. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  6. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  7. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  8. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  9. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
  10. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  11. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 121
  12. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  13. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  14. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 58
  15. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  16. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  17. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 11.

18 papers.

Sort:

By area:

  1. 2025AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understandingcited by 15
  2. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  3. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generationcited by 5
  4. 2025Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTScited by 5
  5. 2026WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Trainingcited by 3
  6. 2025EncouRAGe: Evaluating RAG Local, Fast, and Reliablecited by 2
  7. 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agentscited by 2
  8. 2026Event-Grounded Question Answering over Long Audio via Structured Retrievalcited by 2
  9. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  10. 2026Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Modelscited by 1
  11. 2026Audio Jailbreaks in Large Audio-Language Models: Taxonomy, Attack-Defense Analysis, and Cost-Aware Evaluationcited by 1
  12. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
6 more
  1. 2025Expressive Speech Retrieval using Natural Language Descriptions of Speaking Stylecited by 0
  2. 2026SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialnesscited by 0
  3. 2026ATIR: Towards Audio-Text Interleaved Contextual Retrievalcited by 0
  4. 2026JSPG: Dynamic Dictionary Filtering via Joint Semantic-Pinyin-Glyph Retrieval for Chinese Contextual ASRcited by 0
  5. 2026VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrievalcited by 0
  6. 2026AuraLuxMuse: Adaptive Fusion Modeling for Aesthetic Stage Lighting Design with Music and Expert Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  2. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage
  3. 2024Retrieval Augmented End-to-End Spoken Dialog Models
  4. 2025VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering
  5. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systems
  6. 2026PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding
  7. 2025M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
  8. 2023Retrieval-Augmented Text-to-Audio Generation