Persistent Teacher Anchoring for Tool-Using Agents

arXiv 2609.04773 ↗First posted 4 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Distillation is common in LLM post-training, where on-policy knowledge distillation (OPKD) uses student-generated trajectories to prepare the student for downstream RL. At each state, the student matches a next-token distribution supplied by the teacher. As the rollout enters states the teacher would not visit, the teacher-student distribution gap can accumulate. In tool use, this gap becomes consequential because student-written calls execute before supervision and their observations shape later prefixes. Proposer-verifier generation addresses this drift by letting the teacher decide which student-proposed text is retained during generation. Existing formulations govern text but leave tool execution outside their scope. We propose Persistent Teacher Anchoring (PTA), a student-induced but teacher-committed rollout construction. PTA retains chunk-level verification and adds turn-level commitment, allowing a call to reach the environment only after the teacher has verified the entire turn. Treating verified chunks as atomic generation units, we introduce persistent lookahead, which fills idle rollout capacity by advancing future samples and carrying unfinished ones across student updates under the fixed verifier. Across Search-R1-style retrieval and DeepEyes-style perception RL, applying PTA before downstream RL improves macro best@4 by 2.5 and 2.8 points over OPKD under the same downstream RL budget, while lookahead improves throughput by 24%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 29 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

21 of 29 references.

Closest ancestors · 1

Same direction, newest first

2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs12 Sep

Other references · 20

2017Proximal Policy Optimization Algorithmscited by 14,441
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
2015Unifying distillation and privileged informationcited by 218
2023Speculative Decoding with Big Little Decodercited by 163

8 more

2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
2024FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillationcited by 11
2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Modelscited by 8
2025Fine-tuning with RAG for Improving LLM Learning of New Skillscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way