From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
Recent work on LLM agents is shifting from external capability elicitation to capability internalization, enabling agents to retain useful skills without retrieval at inference time. On-policy self-distillation (OPSD) offers a promising direction, but many existing methods typically supervise students by scoring actions along student-generated trajectories. Such supervision has two limitations: teacher preferences are not validated by environment outcomes, and action-level scores underuse information from student rollouts, teacher rollouts, and their behavioral relationship. We therefore advocate outcome-verified teacher supervision and comparative learning over teacher-student trajectories. Based on this view, we propose Outcome-Verified Comparative Self-Distillation (OVCSD). OVCSD organizes failed student rollouts into a prefix tree, adaptively invokes a skill-conditioned teacher from student-reached states, and retains only outcome-verified successful continuations. It then applies localized comparative learning at the first state-aligned divergence and distills the post-divergence teacher suffix to transfer completion behavior. Experiments on ALFWorld and WebShop across three model scales show that OVCSD consistently outperforms skill-free RL and existing self-distillation baselines, achieving up to 29.7 and 5.4 absolute success-rate gains over the strongest baselines on ALFWorld and WebShop, respectively, while adding less than 3% privileged interaction during training.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 35 references that are in the map. First posted on arXiv 30 Jul 2026.
Builds on
34 of 35 references.
Closest ancestors · 7
Same direction, newest first
- 2026Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents19 May
- 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
- 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System2 Feb
- 2025Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections16 Dec
- 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs12 Sep
- 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
- 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
Other references · 27
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
- 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
- 2025Memory in the Age of AI Agentscited by 285
15 more
- 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
- 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 252
- 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
- 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
- 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
- 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 82
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
- 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 42
- 2025ATLaS: Agent Tuning via Learning Critical Stepscited by 29
- 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
- 2025LIMI: Less is More for Agencycited by 17
- 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
- 2026SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topologycited by 8
- 2026Skill-Conditioned Gated Self-Distillation for LLM Reasoningcited by 6
- 2026SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Trainingcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teachers
- 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
- 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
- 2026Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation
- 2026ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation
- 2026Reward-Aligned Reweighting for On-Policy Distillation
- 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
- 2026PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation