# ICPO: Illocution-Calibrated Policy Optimization for Multi-Turn Conversation

arXiv: https://arxiv.org/abs/2601.15330 · HTML: https://mlsanity.com/paper/2601.15330/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 27 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 26 of 26)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization](https://mlsanity.com/paper/2508.00222/) (2025)
- [On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots](https://mlsanity.com/paper/2406.01633/) (2024)
- [Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends](https://mlsanity.com/paper/2508.11548/) (2025)
- [EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint](https://mlsanity.com/paper/2509.03058/) (2025)
- [CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor](https://mlsanity.com/paper/2509.09703/) (2025)
- [MEraser: An Effective Fingerprint Erasure Approach for Large Language Models](https://mlsanity.com/paper/2506.12551/) (2025)
- [PREE: Towards Harmless and Adaptive Fingerprint Editing in Large Language Models via Knowledge Prefix Enhancement](https://mlsanity.com/paper/2509.00918/) (2025)
- [Fingerprint Vector: Enabling Scalable and Efficient Model Fingerprint Transfer via Vector Addition](https://mlsanity.com/paper/2409.08846/) (2024)
- [AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing](https://mlsanity.com/paper/2601.11007/) (2026)
- [SRAF: Stealthy and Robust Adversarial Fingerprint for Copyright Verification of Large Language Models](https://mlsanity.com/paper/2505.06304/) (2025)
- [Unlocking the Effectiveness of LoRA-FP for Seamless Transfer Implantation of Fingerprints in Downstream Models](https://mlsanity.com/paper/2509.00820/) (2025)
- [Spectral Logit Sculpting: Adaptive Low-Rank Logit Transformation for Controlled Text Generation](https://mlsanity.com/paper/2509.25204/) (2025)
- [ForgetMark: Stealthy Fingerprint Embedding via Targeted Unlearning in Language Models](https://mlsanity.com/paper/2601.08189/) (2026)

## Built on by (most cited first, 2 of 2)

- [MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation](https://mlsanity.com/paper/2605.27186/) (2026)
- [Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization](https://mlsanity.com/paper/2606.27025/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) in multi-turn conversations often suffer from a ``lost-in-conversation'' phenomenon, where they struggle to recover from early incorrect assumptions, particularly when users provide ambiguous initial instructions. We find that standard post-training techniques like Reinforcement Learning with Verifiable Rewards (RLVR) exacerbate this issue by rewarding confident, direct answers, thereby inducing overconfidence and discouraging the model from seeking clarification. To address this, we propose Illocution-Calibrated Policy Optimization (ICPO), a novel training framework that sensitizes the model to instruction ambiguity. ICPO augments the training corpus with underspecified prompts and conditions the reward signal on the user's illocutionary intent, rewarding the model for expressing uncertainty or asking for clarification when faced with ambiguity. Experiments demonstrate that ICPO fosters appropriate humility, yielding a substantial average improvement of 75% in multi-turn conversation, while preserving robust performance on single-turn benchmarks. Our work presents a practical path toward more robust and collaborative conversational AI that can better navigate the nuances of human interaction.