mlsanityData to 14 Oct 2026

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

arXiv 2609.29051 ↗First posted 24 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents also fits: Agent skill learning and evolution

also in 1 more
Abstract (arXiv)

On-policy self-distillation (OPSD) has become a popular recipe for post-training LLM agents. It supervises the agent model at the token level with a stronger teacher view of the same model, obtained by conditioning on privileged information (PI). In this work, we show that in multi-turn agents, this paradigm teaches the student to act with confidence but without the information behind it. The trained agent behaves as if it had privileged information it never observed, and its performance falls well short of plain RL, in the worst case below the untrained base model. Therefore, we propose Privileged Self-Practice (PSP), which keeps the PI and moves it from the loss to the sampler. When the student's rollouts on a task mostly fail, we inject a short per-task instruction written by an analyzer model, sample the task again with the instruction in context, and train on the result with an unchanged GRPO objective. The privileged information stays in the prompt and never enters the loss. Across AppWorld and SWE-bench Verified, with three different student models, PSP obtains the best average score in every setting and is the only method that consistently outperforms plain GRPO, improving task-goal completion by up to 65% on AppWorld and the resolved rate by up to 61% on SWE-bench Verified.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 51 of its 54 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

51 of 54 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning25 Jun
  2. 2026Self-Distilled Agentic Reinforcement Learning14 May
  3. 2026Learning Agentic Policy from Action Guidance12 May
  4. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agents8 May
  5. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate2 May
  6. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  7. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models10 Apr
  8. 2026Hindsight Credit Assignment for Long-Horizon LLM Agents7 Mar
  9. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent20 Nov
  10. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  11. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks19 Mar
  12. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb
1 more
  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
26 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  2. 2016Sequence-Level Knowledge Distillationcited by 801
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  5. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  6. 2026Reinforcement Learning via Self-Distillationcited by 332
  7. 2025Learning to Reason under Off-Policy Guidancecited by 281
  8. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  9. 2026Gemma 4 Technical Reportcited by 259
  10. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  11. 2026MiMo-V2-Flash Technical Reportcited by 191
  12. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  13. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 165
  14. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  15. 2022Learning by Distilling Contextcited by 138
  16. 2026Self-Distilled RLVRcited by 131
  17. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
  18. 2019Distilling Policy Distillationcited by 84
  19. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58
  20. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  21. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  22. 2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillationcited by 31
  23. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 27
  24. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 16
  25. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  26. 2026Learning, Fast and Slow: Towards LLMs That Adapt Continuallycited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
  2. 2026Privileged Information Distillation for Language Models
  3. 2026When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents
  4. 2026ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation
  5. 2026Latent On-Policy Self-Distillation
  6. 2026EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation
  7. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
  8. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation