# ESCRAG-R1: Retrieval-Augmented Reinforcement Learning for Emotional Support Conversation

arXiv: https://arxiv.org/abs/2608.21925 · HTML: https://mlsanity.com/paper/2608.21925/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 22 Aug 2026.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)
- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 18 of 18)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations](https://mlsanity.com/paper/2311.00273/) (2023)
- [CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling](https://mlsanity.com/paper/2405.16433/) (2024)
- [Building Emotional Support Chatbots in the Era of LLMs](https://mlsanity.com/paper/2308.11584/) (2023)
- [HealMe: Harnessing Cognitive Reframing in Large Language Models for Psychotherapy](https://mlsanity.com/paper/2403.05574/) (2024)
- [ESCoT: Towards Interpretable Emotional Support Dialogue Systems](https://mlsanity.com/paper/2406.10960/) (2024)
- [CARE: Commonsense-Aware Emotional Response Generation with Latent Concepts](https://mlsanity.com/paper/2012.08377/) (2020)
- [Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling](https://mlsanity.com/paper/2505.15715/) (2025)
- [Chain of Strategy Optimization Makes Large Language Models Better Emotional Supporter](https://mlsanity.com/paper/2503.05362/) (2025)
- [APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation](https://mlsanity.com/paper/2407.21048/) (2024)
- [IntentionESC: An Intention-Centered Framework for Enhancing Emotional Support in Dialogue Systems](https://mlsanity.com/paper/2506.05947/) (2025)
- [SouLLMate: An Application Enhancing Diverse Mental Health Support with Adaptive LLMs, Prompt Engineering, and RAG Techniques](https://mlsanity.com/paper/2410.16322/) (2024)
- [MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognition](https://mlsanity.com/paper/2507.18929/) (2025)
- [DiaCBT: A Long-Periodic Dialogue Corpus Guided by Cognitive Conceptualization Diagram for CBT-based Psychological Counseling](https://mlsanity.com/paper/2509.02999/) (2025)
- [Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection](https://mlsanity.com/paper/2511.17587/) (2025)

## Abstract (arXiv)

Emotional Support Conversation (ESC) systems aim to provide holistic support by balancing professional therapeutic competence with natural empathy. However, existing methods struggle to simultaneously achieve structured, stage-aware reasoning and seamless empathy-expertise alignment, often resulting in an artificial splicing of clinical strategies and generic reassurance. To overcome these limitations, we propose ESCRAG-R1, a unified framework that integrates retrieval-based psychological guidance into Group Relative Policy Optimization (GRPO). By incorporating retrieval into the reinforcement learning loop, ESCRAG-R1 transforms external knowledge into a robust learning signal that stimulates explicit internal reasoning prior to generation and fundamentally reshapes the model's internal policy. To provide the reliable supervision required for this optimization, we construct ESC-Preference, a high-quality dataset based on a Client--Counselor--Judge evaluation framework that delivers precise, empathy-aware reward signals. Extensive experiments demonstrate that ESCRAG-R1 significantly outperforms existing baselines by mitigating superficial splicing and realizing a natural integration of professional guidance and empathetic expression. Code and datasets are released at https://github.com/Matcha-Liu/ESCRAG-R1.