mlsanityData to 14 Oct 2026

PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning

arXiv 2609.36642 ↗First posted 29 Sep 2026cs.LGcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. Invisible Advantage: a skill in context lifts WebShop success from 42.2% to 56.2%, yet changes the probabilities of fewer than a quarter of the sampled tokens. Much to Align: a skill changes the hidden states of over 80% of response tokens, in a way that linear probes can trace back to the specific skill. To exploit this, we propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective, with no external skill library, separate teacher, or inference overhead. On ALFWorld and WebShop with two backbones, PR-OPD achieves the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at https://github.com/balibata/PR-OPD.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 46 of its 47 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

46 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents10 Aug
  2. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning6 Aug
  3. 2026PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning3 Aug
  4. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning16 Jul
  5. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning25 Jun
  6. 2026Self-Distilled Agentic Reinforcement Learning14 May
  7. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  8. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  9. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 36

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  12. 2016Understanding intermediate layers using linear classifier probescited by 1,353
24 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  7. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  8. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  9. 2019Learning by Cheatingcited by 383
  10. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  11. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 278
  12. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  13. 2015Unifying distillation and privileged informationcited by 233
  14. 2022Learning by Distilling Contextcited by 138
  15. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 94
  16. 2026Privileged Information Distillation for Language Modelscited by 90
  17. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  18. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 35
  19. 2026On the Geometry of On-Policy Distillationcited by 19
  20. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 12
  21. 2026OPRD: On-Policy Representation Distillationcited by 11
  22. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  23. 2026SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolutioncited by 5
  24. 2026LastOPD: Taming Collapse in Latent On-Policy Distillationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Latent-MOPD: Latent Multi-Teacher On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PreSO: Privileged Representation Isolation for On-Policy Self-Distillation
  2. 2026From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents
  3. 2026Latent On-Policy Self-Distillation
  4. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  5. 2026Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
  6. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
  7. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
  8. 2026PHF: Privileged Hidden Flow for On-Policy Self-Distillation