# Convert Language Model into a Value-based Strategic Planner

arXiv: https://arxiv.org/abs/2505.06987 · HTML: https://mlsanity.com/paper/2505.06987/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 11 May 2025.

## Builds on (most cited first, 8 of 8)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL](https://mlsanity.com/paper/2402.19446/) (2024)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation](https://mlsanity.com/paper/2402.13211/) (2024)
- [TransESC: Smoothing Emotional Support Conversation via Turn-Level State Transition](https://mlsanity.com/paper/2305.03296/) (2023)
- [Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Planner](https://mlsanity.com/paper/2406.11978/) (2024)
- [FSM: A Finite State Machine Based Zero-Shot Prompting Paradigm for Multi-Hop Question Answering](https://mlsanity.com/paper/2407.02964/) (2024)

## Built on by (most cited first, 5 of 5)

- [Towards Open-Ended Emotional Support Conversations in LLMs via Reinforcement Learning with Future-Oriented Rewards](https://mlsanity.com/paper/2508.12935/) (2025)
- [PRCCF: A Persona-guided Retrieval and Causal-aware Cognitive Filtering Framework for Emotional Support Conversation](https://mlsanity.com/paper/2604.01671/) (2026)
- [Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS](https://mlsanity.com/paper/2606.09837/) (2026)
- [ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents](https://mlsanity.com/paper/2608.21969/) (2026)
- [FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation](https://mlsanity.com/paper/2610.20051/) (2026)

## Abstract (arXiv)

Emotional support conversation (ESC) aims to alleviate the emotional distress of individuals through effective conversations. Although large language models (LLMs) have obtained remarkable progress on ESC, most of these studies might not define the diagram from the state model perspective, therefore providing a suboptimal solution for long-term satisfaction. To address such an issue, we leverage the Q-learning on LLMs, and propose a framework called straQ*. Our framework allows a plug-and-play LLM to bootstrap the planning during ESC, determine the optimal strategy based on long-term returns, and finally guide the LLM to response. Substantial experiments on ESC datasets suggest that straQ* outperforms many baselines, including direct inference, self-refine, chain of thought, finetuning, and finite state machines.