# Towards Open-Ended Emotional Support Conversations in LLMs via Reinforcement Learning with Future-Oriented Rewards

arXiv: https://arxiv.org/abs/2508.12935 · HTML: https://mlsanity.com/paper/2508.12935/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [MISC: A MIxed Strategy-Aware Model Integrating COMET for Emotional Support Conversation](https://mlsanity.com/paper/2203.13560/) (2022)
- [Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents](https://mlsanity.com/paper/2311.00262/) (2023)
- [Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning](https://mlsanity.com/paper/2305.13660/) (2023)
- [Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning](https://mlsanity.com/paper/2210.04242/) (2022)
- [Control Globally, Understand Locally: A Global-to-Local Hierarchical Graph Network for Emotional Support Conversation](https://mlsanity.com/paper/2204.12749/) (2022)
- [Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation](https://mlsanity.com/paper/2402.13211/) (2024)
- [ESCoT: Towards Interpretable Emotional Support Dialogue Systems](https://mlsanity.com/paper/2406.10960/) (2024)
- [Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversations](https://mlsanity.com/paper/2305.10172/) (2023)
- [PAL: Persona-Augmented Emotional Support Conversation Generation](https://mlsanity.com/paper/2212.09235/) (2022)
- [Planning Like Human: A Dual-process Framework for Dialogue Planning](https://mlsanity.com/paper/2406.05374/) (2024)
- [Controllable Mixed-Initiative Dialogue Generation through Prompting](https://mlsanity.com/paper/2305.04147/) (2023)
- [Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models](https://mlsanity.com/paper/2305.17878/) (2023)
- [TransESC: Smoothing Emotional Support Conversation via Turn-Level State Transition](https://mlsanity.com/paper/2305.03296/) (2023)
- [Chain of Strategy Optimization Makes Large Language Models Better Emotional Supporter](https://mlsanity.com/paper/2503.05362/) (2025)
- [EmoDynamiX: Emotional Support Dialogue Strategy Prediction by Modelling MiXed Emotions and Discourse Dynamics](https://mlsanity.com/paper/2408.08782/) (2024)
- [Knowledge-enhanced Memory Model for Emotional Support Conversation](https://mlsanity.com/paper/2310.07700/) (2023)
- [SocialSim: Towards Socialized Simulation of Emotional Support Conversation](https://mlsanity.com/paper/2506.16756/) (2025)
- [Convert Language Model into a Value-based Strategic Planner](https://mlsanity.com/paper/2505.06987/) (2025)
- [DialogXpert: Driving Intelligent and Emotion-Aware Conversations through Online Value-Based Reinforcement Learning with LLM Priors](https://mlsanity.com/paper/2505.17795/) (2025)

## Built on by (most cited first, 5 of 5)

- [MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue](https://mlsanity.com/paper/2603.06194/) (2026)
- [EmoLLM: Appraisal-Grounded Cognitive-Emotional Co-Reasoning in Large Language Models](https://mlsanity.com/paper/2603.16553/) (2026)
- [Why It Hurts: Identifying the Drivers of Negative Thoughts in Emotional Support Conversations](https://mlsanity.com/paper/2607.28648/) (2026)
- [Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation](https://mlsanity.com/paper/2609.14648/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)

## Abstract (arXiv)

Emotional Support Conversation (ESC) systems aim to alleviate users' emotional difficulties and provide long-term, systematic support for emotional well-being. However, most large language model (LLM)-based ESC systems rely on predefined strategies, which limits their effectiveness in complex, real-life scenarios. To enable flexible responses to diverse emotional problem scenarios, this paper introduces a novel end-to-end framework (RLFF-ESC) that directly learns enduring emotionally supportive response skills using reinforcement learning. For sustained emotional support, we first employ an LLM-based multi-agent mechanism to simulate future dialogue trajectories and collect future-oriented rewards. We then train a future-oriented reward model, which is subsequently used to train the emotional support policy model. Additionally, we incorporate an explicit reasoning process during response generation to further enhance the quality, relevance, and contextual appropriateness of the system's responses. We evaluate the backbone policy model on Qwen2.5-7B-Instruct-1M and LLaMA3.1-8B-Instruct models, testing the proposed RLFF-ESC framework across two public ESC datasets. Experimental results demonstrate that RLFF-ESC consistently outperforms existing baselines in terms of goal completion and response quality.