# ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation

arXiv: https://arxiv.org/abs/2606.23104 · HTML: https://mlsanity.com/paper/2606.23104/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning](https://mlsanity.com/paper/2506.01347/) (2025)
- [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://mlsanity.com/paper/2605.00674/) (2026)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling](https://mlsanity.com/paper/2410.11325/) (2024)
- [DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs](https://mlsanity.com/paper/2503.07067/) (2025)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe](https://mlsanity.com/paper/2605.03677/) (2026)
- [SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting](https://mlsanity.com/paper/2604.10688/) (2026)
- [GATES: Self-Distillation under Privileged Context with Consensus Gating](https://mlsanity.com/paper/2602.20574/) (2026)
- [Efficient Knowledge Injection in LLMs via Self-Distillation](https://mlsanity.com/paper/2412.14964/) (2024)
- [Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty](https://mlsanity.com/paper/2603.15500/) (2026)

## Built on by (most cited first, 7 of 7)

- [What Does Privileged Information Add to On-Policy Self-Distillation?](https://mlsanity.com/paper/2609.20612/) (2026)
- [Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers](https://mlsanity.com/paper/2609.18321/) (2026)
- [Leaky Students: Membership Inference against On-Policy Distillation](https://mlsanity.com/paper/2609.33136/) (2026)
- [What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillation](https://mlsanity.com/paper/2609.33455/) (2026)
- [Dual-Vocabulary Language Model for Cross-Tokenizer Distillation](https://mlsanity.com/paper/2609.33816/) (2026)
- [Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation](https://mlsanity.com/paper/2609.39687/) (2026)
- [Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation](https://mlsanity.com/paper/2610.02678/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) improves LLM reasoning by training a student model on its own generated outputs, but standard OPD treats all student-generated outputs (SGOs) equally regardless of their informativeness. We observe a consistent asymmetry in controlled filtering experiments: in both OPD and on-policy self distillation (OPSD), training only on incorrect SGOs outperforms training only on correct ones. Our further analysis suggests that models trained on correct-only SGOs tend to generate shorter reasoning traces and show weaker reflection behavior, while incorrect SGOs better preserve exploratory reasoning near the model's capability boundary. To exploit this signal without requiring full answer-containing rollouts, we introduce ReNIO, which Reweights Negative trajectory Importance for LLM On-policy distillation. By using the student-to-teacher probability ratio, ReNIO identifies pivotal tokens leading to wrong reasoning traces and aggregates their information into a normalized sample weight, inherently assigning larger weights to likely negative trajectories without observing the correctness of final-answer. Since Re-NIO only uses prefix-conditioned token probabilities, it preserves OPD's prefix training advantage over full-rollout reinforcement learning. Across both mathematical reasoning and code generation tasks, ReNIO improves both OPD and OPSD, with representative relative gains of up to 8.90% for Qwen3-1.7B and 10.00% for R1-Distill-Qwen-7B on mathematical reasoning benchmarks. Code repo: https://github.com/BDML-lab/ReNIO.