mlsanityData to 14 Oct 2026

Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation

arXiv 2606.02684 ↗First posted 1 Jun 2026cs.LG cs.AI cs.CLcited by 32

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

On-Policy distillation (OPD) in large language models is shifting from full-trace KL supervision toward more selective training paradigms. Recent OPD methods increasingly focus on selecting which trajectories to learn from, which tokens are most informative, and which supervision signals are most reliable. Motivated by this trend, we rethink optimization granularity of OPD and propose fireicon FiRe-OPD (Filter, then Reweight), which jointly adjusts supervision signals at both trajectory and token levels. In details, FiRe-OPD first filters trajectories to remove low-quality rollout samples, and then applies soft reweighting within the retained trajectories to emphasize informative tokens. Compared with hard token selection, FiRe-OPD leverages a soft-weighting mechanism to effectively mitigate information loss and enhance optimization stability, thereby achieving finer-grained OPD optimization. We validate the effectiveness of FiRe-OPD across strong-to-weak, single-teacher, and multi-teacher settings, and demonstrate its superiority over recent token-level OPD methods ( (e.g., +6.25 on AIME 2024 in strong-to-weak, +18.81 on Miner in multi-teacher). Our code is available at https://github.com/YuYingLi0/FiRe-OPD.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 48 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

35 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents27 Apr
  2. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  3. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation14 Apr
  4. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting12 Apr
  5. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  6. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation11 Mar
  7. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoning26 Feb
  8. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 27

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2026Reinforcement Learning via Self-Distillationcited by 332
  5. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  6. 2026On-Policy Context Distillation for Language Modelscited by 186
  7. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  8. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  9. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  10. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  11. 2026Self-Distilled RLVRcited by 131
  12. 2026TIP: Token Importance in On-Policy Distillationcited by 104
15 more
  1. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  2. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 53
  3. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 53
  4. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  5. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  6. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 36
  7. 2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillationcited by 31
  8. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 30
  9. 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Modelscited by 30
  10. 2025pi-Flow: Policy-Based Few-Step Generation via Imitation Distillationcited by 29
  11. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 28
  12. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 16
  13. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillationcited by 13
  14. 2026Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Trainingcited by 11
  15. 2026Hybrid Policy Distillation for LLMscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 32.

32 papers.

Sort:

By area:

  1. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 10
  2. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 6
  3. 2026What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selectioncited by 6
  4. 2026When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillationcited by 5
  5. 2026When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillationcited by 4
  6. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 4
  7. 2026Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoningcited by 4
  8. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillationcited by 3
  9. 2026ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillationcited by 3
  10. 2026Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoningcited by 2
  11. 2026D3-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillationcited by 2
  12. 2026SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactionscited by 2
20 more
  1. 2026Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillationcited by 2
  2. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  3. 2026Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillationcited by 1
  4. 2026Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewardscited by 1
  5. 2026Blockwise Policy-Drift Gating for On-Policy Distillationcited by 0
  6. 2026AlphaG-OPD: Reliability-Gated Sibling Counterfactuals for On-Policy Distillation in Symbolic Alpha Factor Discoverycited by 0
  7. 2026SciLENS: RL-Driven Autonomous Agents for Scientific Localized Evidence Navigation and Synthesiscited by 0
  8. 2026KuaiRP Series Role-playing Models Technical Reportcited by 0
  9. 2026Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teacherscited by 0
  10. 2026Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillationcited by 0
  11. 2026What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillationcited by 0
  12. 2026DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillationcited by 0
  13. 2026Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillationcited by 0
  14. 2026On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamicscited by 0
  15. 2026Reward-Aligned Reweighting for On-Policy Distillationcited by 0
  16. 2026REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reusecited by 0
  17. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0
  18. 2026DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillationcited by 0
  19. 2026MetaOPD: Meta-Learned Token Weighting for On-Policy Distillationcited by 0
  20. 2026Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
  2. 2026Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
  3. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  4. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Example
  5. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
  6. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  7. 2026Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
  8. 2026Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation