# Teach Me After You Learn It: Lookahead Self-Distillation for RLVR

arXiv: https://arxiv.org/abs/2610.18256 · HTML: https://mlsanity.com/paper/2610.18256/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 26 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 24 of 24)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [Black-Box On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2511.10643/) (2025)
- [RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation](https://mlsanity.com/paper/2606.11709/) (2026)
- [Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation](https://mlsanity.com/paper/2604.13010/) (2026)
- [d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models](https://mlsanity.com/paper/2512.09675/) (2025)
- [Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning](https://mlsanity.com/paper/2605.11458/) (2026)
- [Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization](https://mlsanity.com/paper/2605.05040/) (2026)
- [Self-Policy Distillation via Capability-Selective Subspace Projection](https://mlsanity.com/paper/2605.22675/) (2026)

## Abstract (arXiv)

On-policy self-distillation (OPSD) has emerged as a popular training paradigm for LLMs, providing dense, token-level supervision by aligning a model's own distribution with the distribution induced by privileged information. However, existing OPSD methods derive supervision solely from privileged information of the same instance, leaving inter-instance supervision largely unexplored. We propose Lookahead Self-Distillation (LEAD), which introduces inter-instance distillation by allowing the teacher to first learn from other on-policy rollouts before teaching the student. LEAD constructs lookahead teachers through virtual RL updates on disjoint subsets of on-policy rollouts, and uses each teacher to provide cross-group token-level supervision for the other subset. To reconcile the complementary yet potentially conflicting signals from OPSD and RLVR, we introduce harmonious modulation, which adaptively modulates the GRPO advantage based on the agreement between the two signals. Extensive experiments on scientific reasoning and interactive-agent benchmarks show that LEAD achieves the best in-domain average performance across model scales while maintaining out-of-domain generalization comparable to GRPO.