mlsanityData to 7 Oct 2026

From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL

arXiv 2609.39436 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning with a verifiable reward (RLVR) offers a scalable approach to training language-model agents, yet sparse outcome rewards can leave early training with little signal for policy improvement. We identify an On-Policy Acceleration Phenomenon: in our main comparisons, RLVR initialized with on-policy distillation reaches high performance earlier in training and achieves both higher average performance during subsequent RLVR and higher final performance than the alternative baselines. Motivated by this observation, we study On-Policy Warmup (OPW), a teacher-guided stage in which the student trains with teacher supervision on its own interaction trajectories before transitioning to RLVR. Unlike imitation on fixed teacher-generated trajectories, OPW targets states induced by the student's own decisions, including imperfect actions and recovery situations. We provide a theoretical explanation by connecting on-policy reverse-KL distillation to trajectory-level distribution matching. Under a competent teacher and sufficiently small population distillation loss, this connection yields a lower bound on initial verifier success and a corresponding bound on reward-discovery complexity. For group-relative RLVR, we further characterize when increased success probability produces more reward-informative groups. Together, our findings support on-policy distillation as an effective warmup for agentic RLVR and identify initial reward discovery as a mechanism that can contribute to the observed acceleration.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 28 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

24 of 28 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performance26 Aug
  2. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning16 Jul
  3. 2026OpenClaw-RL: Train Any Agent Simply by Talking10 Mar

Other references · 21

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  8. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  9. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  10. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  11. 2025Learning to Reason under Off-Policy Guidancecited by 266
  12. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
9 more
  1. 2025The Coverage Principle: How Pre-Training Enables Post-Trainingcited by 25
  2. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 24
  3. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  4. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
  5. 2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8
  6. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 7
  7. 2025Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settingscited by 3
  8. 2026OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoningcited by 3
  9. 2026RL Starts before RL: On Policy Distillation for Better Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On-policy Distillation with Verifiable Reward
  2. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  3. 2026Decoupling Exploration from Optimization in RLVR
  4. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
  5. 2025Refined Policy Distillation: From VLA Generalists to RL Experts
  6. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholds
  7. 2026Group-Reflective Self-Distillation for Agentic Reinforcement Learning
  8. 2026Rewards as Labels: Revisiting RLVR from a Classification Perspective