# MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue

arXiv: https://arxiv.org/abs/2603.06194 · HTML: https://mlsanity.com/paper/2603.06194/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 51 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 30 of 44)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent](https://mlsanity.com/paper/2507.02259/) (2025)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support](https://mlsanity.com/paper/2009.08441/) (2020)
- [WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2505.16421/) (2025)
- [MIME: MIMicking Emotions for Empathetic Response Generation](https://mlsanity.com/paper/2010.01454/) (2020)
- [SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations](https://mlsanity.com/paper/2311.00273/) (2023)
- [Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools](https://mlsanity.com/paper/2502.04644/) (2025)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment](https://mlsanity.com/paper/2505.11821/) (2025)
- [Control Globally, Understand Locally: A Global-to-Local Hierarchical Graph Network for Emotional Support Conversation](https://mlsanity.com/paper/2204.12749/) (2022)
- [EmoBench: Evaluating the Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2402.12071/) (2024)
- [SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Support](https://mlsanity.com/paper/2305.00450/) (2023)
- [Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements](https://mlsanity.com/paper/2310.05140/) (2023)
- [Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents](https://mlsanity.com/paper/2509.23040/) (2025)
- [Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances](https://mlsanity.com/paper/2407.21315/) (2024)
- [Exploiting Tree Structure for Credit Assignment in RL Training of LLMs](https://mlsanity.com/paper/2509.18314/) (2025)

## Built on by (most cited first, 3 of 3)

- [The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents](https://mlsanity.com/paper/2608.06663/) (2026)
- [Evolving Support Priorities in Empathetic Reinforcement Learning](https://mlsanity.com/paper/2609.34249/) (2026)
- [PsyEvo: A Personalized Counseling Agent That Self-Evolves at Test Time](https://mlsanity.com/paper/2610.02885/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) for large language models (LLMs) has shown strong performance in single-turn tasks, but extending it to multi-turn interaction remains challenging due to sparse rewards and poor per-turn credit assignment. In emotional support dialogues, responses shape future user states, so matched-state step-wise comparison is unavailable, while trajectory-level supervision is insufficient. We propose MICA (Multi-granularity Intertemporal Credit Assignment), a critic-free RL framework for multi-turn emotional support tasks. MICA derives both immediate and delayed credit from a shared potential function over the user's structured support state. Incremental Distance Reward measures the per-turn decrease in residual distance to the target state, while its Monte Carlo return captures delayed effects. After scope-specific normalization, the two signals form a mixed advantage for stable per-turn optimization without matched-state comparisons, rollout trees, or a learned critic. On EMPA, EQ-Bench, and EmoBench with Qwen2.5-7B-Instruct and Qwen3-8B/14B/32B, MICA consistently outperforms GRPO and REINFORCE++, achieving up to +42.5 on EMPA, while adding no rollout cost and remaining robust to reward judges. These results show that turn-aware credit assignment enables effective and practical multi-turn RL for interactive LLMs.