Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
On-policy self-distillation (OPSD) gives language agents dense token-level supervision from a privileged self-teacher on the policy's own trajectories. Existing methods allocate this supervision mainly by teacher trust, but trust does not reveal whether emphasizing a token supports the current policy objective. We call this the trust-utility mismatch and introduce Influence Calibration for Self-Distillation (ICSD). For each supervised token, ICSD measures the first-order response of its importance-weighted RL surrogate contribution to a teacher-directed output perturbation. Batch-adaptive calibration converts this non-stationary signal into a bounded allocation weight while preserving the original auxiliary-loss mass within each action turn. These detached weights affect only the distillation loss and require no additional model pass. Across ALFWorld, WebShop, and Search-QA, ICSD improves all matched aggregate metrics over trust-only allocation under Group Relative Policy Optimization (GRPO) and Group-in-Group Policy Optimization (GiGPO), across two model families spanning 1.5B to 7B. At 7B, it reaches 96.1% ALFWorld success and a WebShop score of 93.1. Frozen-batch analyses show that ICSD reduces teacher-supported mass assigned to objective-opposed tokens from 60.1% to 37.8% and raises cosine compatibility with the RL gradient by 0.192. A companion repository is avail- able at https://github.com/lanqz7766/Influence-Calibration-for-On-Policy-Self-Distillation-in-Agentic-RL.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 26 references that are in the map. First posted on arXiv 14 Aug 2026.
Builds on
24 of 26 references.
Closest ancestors · 5
Same direction, newest first
- 2026CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning28 Jun
- 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation17 Jun
- 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment8 Jun
- 2026Self-Distilled Agentic Reinforcement Learning14 May
- 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
Other references · 19
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2015Trust Region Policy Optimizationcited by 2,503
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
- 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 98
- 2026TIP: Token Importance in On-Policy Distillationcited by 90
- 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
7 more
- 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 28
- 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
- 2026On the Position Bias of On-Policy Distillationcited by 23
- 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 20
- 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Samplingcited by 12
- 2026From Generic Correlation to Input-Specific Credit in On-Policy Self Distillationcited by 9
- 2026DOPD: Dual On-policy Distillationcited by 9
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
- 2026GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents
- 2026Group-Reflective Self-Distillation for Agentic Reinforcement Learning
- 2026PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
- 2026SOD: Step-wise On-policy Distillation for Small Language Model Agents
- 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
- 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
- 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning