mlsanityData to 7 Oct 2026

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

arXiv 2603.11137 ↗First posted 11 Mar 2026cs.LG cs.CLcited by 71

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

On-policy distillation is pivotal for transferring reasoning capabilities to capacity-constrained models, yet remains prone to instability and negative transfer. We show that on-policy distillation can be interpreted, both theoretically and empirically, as a form of policy optimization, where the teacher-student log-likelihood ratio acts as a token reward. From this insight, we introduce REOPOLD (Relaxed On-Policy Distillation) a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation. Specifically, REOPOLD temperately and selectively leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy. Empirically, REOPOLD surpasses its baselines with superior sample efficiency during training and enhanced test-time scaling at inference, across mathematical, visual, and agentic tool-use reasoning tasks. Specifically, REOPOLD outperforms recent RL approaches achieving 6.7~12x greater sample efficiency and enables a 7B student to match a 32B teacher in visual reasoning with a ~3.32x inference speedup.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 70 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

60 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs13 Jun
  2. 2022Large Language Models Are Reasoning Teachers20 Dec
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
38 more
  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2025Group Sequence Policy Optimizationcited by 788
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  8. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  9. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  10. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  11. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  12. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  13. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  14. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  15. 2021InfographicVQAcited by 430
  16. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  17. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  18. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  19. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  20. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  21. 2025Learning to Reason under Off-Policy Guidancecited by 266
  22. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  23. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  24. 2023The False Promise of Imitating Proprietary LLMscited by 254
  25. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  26. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  27. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  28. 2026MiMo-V2-Flash Technical Reportcited by 163
  29. 2018TallyQA: Answering Complex Counting Questionscited by 152
  30. 2025Geometric-Mean Policy Optimizationcited by 107
  31. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  32. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  33. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  34. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  35. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 72
  36. 2023Interpretable Neural-Symbolic Concept Reasoningcited by 56
  37. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 46
  38. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 71.

71 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  3. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 60
  4. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  5. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  6. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 38
  7. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  8. 2026Trust Region On-Policy Distillationcited by 26
  9. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  10. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  11. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 23
  12. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 23
38 more
  1. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  2. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
  3. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  4. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  5. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 14
  6. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillationcited by 11
  7. 2026OPRD: On-Policy Representation Distillationcited by 9
  8. 2026DOPD: Dual On-policy Distillationcited by 9
  9. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  10. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 8
  11. 2026On-Policy Delta Distillationcited by 8
  12. 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidancecited by 8
  13. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  14. 2026PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformationcited by 7
  15. 2026Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rolloutscited by 7
  16. 2026Weak-to-Strong On-Policy Distillationcited by 7
  17. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 7
  18. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillationcited by 6
  19. 2026Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoningcited by 5
  20. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  21. 2026Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillationcited by 5
  22. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillationcited by 5
  23. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scalingcited by 5
  24. 2026CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillationcited by 5
  25. 2026A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillationcited by 5
  26. 2026boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Controlcited by 4
  27. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  28. 2026SAF-OPD: Stable Advantage Fusion for On-Policy Distillationcited by 4
  29. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillationcited by 4
  30. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  31. 2026Simple-OPD: Demystifying Warm-up for On-policy Distillationcited by 3
  32. 2026What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selectioncited by 3
  33. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 3
  34. 2026SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervisioncited by 2
  35. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Trainingcited by 2
  36. 2026Trust Region Policy Distillationcited by 2
  37. 2026DAPD: Dual-Anchored Policy Distillationcited by 2
  38. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
  2. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  3. 2026Reward-Gated On-Policy Distillation
  4. 2026ResOPD: Tail Residualization for Sparse On-Policy Distillation
  5. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  6. 2026Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers
  7. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  8. 2026An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning