mlsanityData to 7 Oct 2026

Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis

arXiv 2501.06467 ↗First posted 11 Jan 2025cs.CLcited by 4

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Conversational speech synthesis (CSS) aims to take the current dialogue (CD) history as a reference to synthesize expressive speech that aligns with the conversational style. Unlike CD, stored dialogue (SD) contains preserved dialogue fragments from earlier stages of user-agent interaction, which include style expression knowledge relevant to scenarios similar to those in CD. Note that this knowledge plays a significant role in enabling the agent to synthesize expressive conversational speech that generates empathetic feedback. However, prior research has overlooked this aspect. To address this issue, we propose a novel Retrieval-Augmented Dialogue Knowledge Aggregation scheme for expressive CSS, termed RADKA-CSS, which includes three main components: 1) To effectively retrieve dialogues from SD that are similar to CD in terms of both semantic and style. First, we build a stored dialogue semantic-style database (SDSSD) which includes the text and audio samples. Then, we design a multi-attribute retrieval scheme to match the dialogue semantic and style vectors of the CD with the stored dialogue semantic and style vectors in the SDSSD, retrieving the most similar dialogues. 2) To effectively utilize the style knowledge from CD and SD, we propose adopting the multi-granularity graph structure to encode the dialogue and introducing a multi-source style knowledge aggregation mechanism. 3) Finally, the aggregated style knowledge are fed into the speech synthesizer to help the agent synthesize expressive speech that aligns with the conversational style. We conducted a comprehensive and in-depth experiment based on the DailyTalk dataset, which is a benchmarking dataset for the CSS task. Both objective and subjective evaluations demonstrate that RADKA-CSS outperforms baseline models in expressiveness rendering. Code and audio samples can be found at: https://github.com/Coder-jzq/RADKA-CSS.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 45 references that are in the map. First posted on arXiv 11 Jan 2025.

Builds on

25 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues15 May
  2. 2024Retrieval-Augmented Audio Deepfake Detection22 Apr
  3. 2023Retrieval-Augmented Text-to-Audio Generation14 Sep

Other references · 22

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2020Supervised Contrastive Learningcited by 2,803
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  9. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  10. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
  11. 2022Control Globally, Understand Locally: A Global-to-Local Hierarchical Graph Network for Emotional Support Conversationcited by 56
  12. 2020Conversational End-to-End TTS for Voice Agentcited by 37
10 more
  1. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 26
  2. 2024Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Checkcited by 24
  3. 2023CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-trainingcited by 17
  4. 2023Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modelingcited by 17
  5. 2024Generative Expressive Conversational Speech Synthesiscited by 16
  6. 2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
  7. 2022FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesiscited by 9
  8. 2024Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretrainingcited by 9
  9. 2023CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesiscited by 8
  10. 2022Explicit Intensity Control for Accented Text-to-speechcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesiscited by 2
  2. 2026MedSpeak: A Knowledge Graph-Aided ASR Error Correction Framework for Spoken Medical QAcited by 1
  3. 2026CoCR-RAG: Enhancing Retrieval-Augmented Generation in Web Q&A via Concept-oriented Context Reconstructioncited by 0
  4. 2026S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation
  2. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systems
  3. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues
  4. 2026AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis
  5. 2022Long-term Control for Dialogue Generation: Methods and Evaluation
  6. 2021Reason first, then respond: Modular Generation for Knowledge-infused Dialogue
  7. 2025APIDA-Chat: Structured Synthesis of API Search Dialogues to Bootstrap Conversational Agents
  8. 2026SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness