# Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections

arXiv: https://arxiv.org/abs/2512.14895 · HTML: https://mlsanity.com/paper/2512.14895/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Cumulative Reasoning with Large Language Models](https://mlsanity.com/paper/2308.04371/) (2023)
- [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024)
- [Improving Code Generation by Training with Natural Language Feedback](https://mlsanity.com/paper/2303.16749/) (2023)
- [Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning](https://mlsanity.com/paper/2410.12952/) (2024)
- [Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards](https://mlsanity.com/paper/2506.11425/) (2025)
- [Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs](https://mlsanity.com/paper/2506.19290/) (2025)
- [Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models](https://mlsanity.com/paper/2506.13923/) (2025)

## Built on by (most cited first, 19 of 19)

- [TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents](https://mlsanity.com/paper/2604.24005/) (2026)
- [Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning](https://mlsanity.com/paper/2608.16333/) (2026)
- [Revisiting DAgger in the Era of LLM-Agents](https://mlsanity.com/paper/2605.12913/) (2026)
- [LLM4Cov: Execution-Aware Agentic Learning for High-coverage Testbench Generation](https://mlsanity.com/paper/2602.16953/) (2026)
- [Hindsight Hint Distillation: Scaffolded Reasoning for SWE Agents from CoT-free Answers](https://mlsanity.com/paper/2605.11556/) (2026)
- [Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails](https://mlsanity.com/paper/2609.09134/) (2026)
- [A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training](https://mlsanity.com/paper/2607.04574/) (2026)
- [Speculative Rollback Correction for Quality-Diverse Web Agent Imitation](https://mlsanity.com/paper/2606.12485/) (2026)
- [Skill-Guided Continuation Distillation for GUI Agents](https://mlsanity.com/paper/2606.18890/) (2026)
- [From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents](https://mlsanity.com/paper/2607.27937/) (2026)
- [Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents](https://mlsanity.com/paper/2605.28775/) (2026)
- [Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift](https://mlsanity.com/paper/2605.09183/) (2026)
- [PUBG Ally: A Conversational Embodied Agent as an AI Teammate](https://mlsanity.com/paper/2609.29837/) (2026)
- [Learning from Teacher Continuations at Student States](https://mlsanity.com/paper/2609.36246/) (2026)
- [Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses](https://mlsanity.com/paper/2609.36630/) (2026)
- [Branching On-Policy Distillation](https://mlsanity.com/paper/2610.13481/) (2026)
- [State Representation Shapes Long-Horizon Agent Learning and Transfer: An Empirical Study](https://mlsanity.com/paper/2610.15407/) (2026)
- [DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision](https://mlsanity.com/paper/2610.19302/) (2026)
- [Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents](https://mlsanity.com/paper/2610.19834/) (2026)

## Abstract (arXiv)

A popular paradigm for training LM agents relies on imitation learning, fine-tuning on expert trajectories. However, we show that the off-policy nature of imitation learning for multi-turn LM agents suffers from the fundamental limitation known as covariate shift: as the student policy's behavior diverges from the expert's, it encounters states not present in the training data, reducing the effectiveness of fine-tuning. Taking inspiration from the classic DAgger algorithm, we propose a novel data generation methodology for addressing covariate shift for multi-turn LLM training. We introduce on-policy expert corrections (OECs), partially on-policy data generated by starting rollouts with a student model and then switching to an expert model part way through the trajectory. We explore the effectiveness of our data generation technique in the domain of software engineering (SWE) tasks, a multi-turn setting where LLM agents must interact with a development environment to fix software bugs. Our experiments compare OEC data against various other on-policy and imitation learning approaches on SWE agent problems and train models using a common rejection sampling (i.e., using environment reward) combined with supervised fine-tuning technique. Experiments find that OEC trajectories show a relative 14% and 13% improvement over traditional imitation learning in the 7b and 32b setting, respectively, on SWE-bench verified. Our results demonstrate the need for combining expert demonstrations with on-policy data for effective multi-turn LM agent training.