mlsanityData to 14 Oct 2026

Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation

arXiv 2505.00028 ↗First posted 27 Apr 2025cs.CL cs.AI cs.IRcited by 5

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

End-to-end speech-to-speech (S2S) dialogue systems have recently garnered increasing research attention for their lower latency and more natural integration of nonverbal cues such as emotion and speaker identity. However, these systems face key challenges, particularly in incorporating external knowledge, a capability commonly addressed by Retrieval-Augmented Generation (RAG) in text-based large language models (LLMs). The core difficulty lies in the modality gap between input speech and retrieved textual knowledge, which hinders effective integration of information. To address this issue, we propose a novel end-to-end RAG framework that directly retrieves relevant textual knowledge from speech queries. Experimental results demonstrate that our method significantly improves the performance of end-to-end S2S dialogue systems while achieving higher retrieval efficiency. Although the overall performance still lags behind the SOTA cascaded models, our framework offers a promising direction for enhancing knowledge integration in end-to-end S2S systems. Our code and dataset are released.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 41 references that are in the map. First posted on arXiv 27 Apr 2025.

Builds on

31 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models20 Feb
  2. 2025SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation26 Jan
  3. 2024Retrieval Augmented End-to-End Spoken Dialog Models2 Feb

Other references · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2017Billion-scale similarity search with GPUscited by 2,497
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  12. 2025Qwen2.5-Omni Technical Reportcited by 736
16 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  3. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  4. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  5. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  6. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  7. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  8. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  9. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  10. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
  11. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  12. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  13. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 62
  14. 2024SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Trainingcited by 59
  15. 2024iRAG: Advancing RAG for Videos with an Incremental Approachcited by 15
  16. 2025Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applicationscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluencycited by 37
  2. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 15
  3. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  4. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
  5. 2026A Reranker for Orchestrating Heterogeneous Speech and Text Retrieverscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  2. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systems
  3. 2024Adaptive Retrieval-Augmented Generation for Conversational Systems
  4. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognition
  5. 2024The 2nd FutureDial Challenge: Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG)
  6. 2022AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialog
  7. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues
  8. 2026RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation