# The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

arXiv: https://arxiv.org/abs/2607.24720 · HTML: https://mlsanity.com/paper/2607.24720/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 49)

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models](https://mlsanity.com/paper/2512.07783/) (2025)
- [Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision](https://mlsanity.com/paper/2604.12002/) (2026)
- [Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation](https://mlsanity.com/paper/2603.19220/) (2026)
- [Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe](https://mlsanity.com/paper/2605.03677/) (2026)
- [TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents](https://mlsanity.com/paper/2604.24005/) (2026)
- [DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints](https://mlsanity.com/paper/2601.18137/) (2026)
- [The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes](https://mlsanity.com/paper/2605.11182/) (2026)
- [From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones](https://mlsanity.com/paper/2509.25123/) (2025)
- [POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration](https://mlsanity.com/paper/2601.18779/) (2026)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis](https://mlsanity.com/paper/2603.20278/) (2026)

## Built on by (most cited first, 3 of 3)

- [Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learning](https://mlsanity.com/paper/2608.03571/) (2026)
- [SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs](https://mlsanity.com/paper/2608.03573/) (2026)
- [Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging](https://mlsanity.com/paper/2609.32303/) (2026)

## Abstract (arXiv)

Multi-turn long-horizon planning is critical for foundation model agents, yet how to fundamentally improve it remains unclear. Existing models are trained on uncontrollable and opaque Internet data, making it difficult to identify how planning ability is acquired, shaped, and integrated. To address this challenge, we introduce a unified and controlled multi-turn environment that enables precise control. It allows systematically study long-horizon planning across three stages. (1) Planning ability acquisition during pre-training. We study data format, distribution, and quality. Explicit world model construction through CoT state transition modeling yields stronger long-horizon generalization. Atomic skills alone are insufficient for compositional generalization, whereas a litte long-horizon data works. Moreover, suboptimal trajectories severely impair performance because errors amplify over long horizons. (2) Planning ability shaping via GRPO and OPD post-training. Through mutual information, we distinguish general planning patterns from task-specific planning knowledge. For planning patterns, we identify three application regions of post-training: unnecessary, effective, and unsupported. OPD has a broader effective region than GRPO under low-quality and long-horizon settings, as it provides more consistent update directions. For planning knowledge, distilling unseen procedures from a teacher with different knowledge may impair student's prior world modeling without fully establishing new knowledge. (3) Planning ability integration through MOPD post-training. We show that multi-teacher on-policy distillation (MOPD) integrates capabilities by converging to shared planning-pattern across environments. Compatible patterns enable cross-environment generalization, partially shared patterns support continual learning, while completely conflicting patterns cause severe interference.