# OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

arXiv: https://arxiv.org/abs/2606.26790 · HTML: https://mlsanity.com/paper/2606.26790/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 38)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization](https://mlsanity.com/paper/2604.02268/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision](https://mlsanity.com/paper/2604.12002/) (2026)
- [Self-Distilled Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.15155/) (2026)

## Built on by (most cited first, 30 of 32)

- [SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.14777/) (2026)
- [Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation](https://mlsanity.com/paper/2608.04794/) (2026)
- [PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.01837/) (2026)
- [Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation](https://mlsanity.com/paper/2608.04788/) (2026)
- [Group-Reflective Self-Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.28076/) (2026)
- [SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment](https://mlsanity.com/paper/2609.02786/) (2026)
- [LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning](https://mlsanity.com/paper/2607.13501/) (2026)
- [LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks](https://mlsanity.com/paper/2607.18110/) (2026)
- [Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization](https://mlsanity.com/paper/2608.31077/) (2026)
- [SKILL-KD: Contrastive Skill Distillation for LLM Agents](https://mlsanity.com/paper/2607.28048/) (2026)
- [MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation](https://mlsanity.com/paper/2607.29320/) (2026)
- [Emotion2Skill: Model-Internal Emotion Signals for Adaptive Skill Selection and Evolution](https://mlsanity.com/paper/2608.09248/) (2026)
- [Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents](https://mlsanity.com/paper/2608.09555/) (2026)
- [Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents](https://mlsanity.com/paper/2609.08919/) (2026)
- [PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.36642/) (2026)
- [From Collaboration to Capability: Internalizing Routed LLM Experts into Compact Reasoners](https://mlsanity.com/paper/2609.12578/) (2026)
- [Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings](https://mlsanity.com/paper/2609.15296/) (2026)
- [Learning What to Activate: Combinatorial Capability Allocation for Long-Horizon Multimodal Agents](https://mlsanity.com/paper/2609.27869/) (2026)
- [From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents](https://mlsanity.com/paper/2609.29051/) (2026)
- [CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning](https://mlsanity.com/paper/2609.29518/) (2026)
- [Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents](https://mlsanity.com/paper/2609.33391/) (2026)
- [TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL](https://mlsanity.com/paper/2609.35058/) (2026)
- [Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teachers](https://mlsanity.com/paper/2609.37522/) (2026)
- [VISTA: Internalizing Collective Visual Experience via On-Policy Distillation for Active Multimodal Agents](https://mlsanity.com/paper/2609.38086/) (2026)
- [EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Making](https://mlsanity.com/paper/2609.38334/) (2026)
- [ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents](https://mlsanity.com/paper/2609.40253/) (2026)
- [PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents](https://mlsanity.com/paper/2609.40285/) (2026)
- [My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.01161/) (2026)
- [GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents](https://mlsanity.com/paper/2610.08959/) (2026)
- [SkillCycle: Co-Evolving Agent Policies and Skill Banks](https://mlsanity.com/paper/2610.09430/) (2026)

## Abstract (arXiv)

Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose OPID (On-Policy Skill Distillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.