mlsanityData to 7 Oct 2026

Learning to Reason under Off-Policy Guidance

arXiv 2504.14945 ↗First posted 21 Apr 2025cs.LG cs.AI cs.CLcited by 266

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~(textitRLVR). However, existing textitRLVR approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. To address this issue, we introduce LUFFY (Learning to reason Under oFF-policY guidance), a framework that augments textitRLVR with off-policy reasoning traces. LUFFY dynamically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. Specifically, LUFFY combines the Mixed-Policy GRPO framework, which has a theoretically guaranteed convergence rate, alongside policy shaping via regularized importance sampling to avoid superficial and rigid imitation during mixed-policy training. Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails. These results provide compelling evidence that LUFFY transcends the fundamental limitations of on-policy RLVR and demonstrates the great potential of utilizing off-policy guidance in RLVR.

Key facts. Cited by 266 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 74 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

43 of 74 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  3. 2025Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning6 Apr
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  7. 2025Process Reinforcement through Implicit Rewards3 Feb
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 33

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
21 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2025s1: Simple test-time scalingcited by 1,311
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  10. 2025LIMO: Less is More for Reasoningcited by 488
  11. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  12. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  13. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  14. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  15. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  16. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  17. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  18. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  19. 2025Concise Reasoning via Reinforcement Learningcited by 78
  20. 2025BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillationcited by 23
  21. 2025Thinking Preference Optimizationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 103; 2026: 163.

266 papers.

Sort:

By area:

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  6. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  7. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  8. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
  9. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 77
  10. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  11. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  12. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
38 more
  1. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  2. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
  3. 2025ExGRPO: Learning to Reason from Experiencecited by 49
  4. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 48
  5. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 40
  6. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 39
  7. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  8. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  9. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 37
  10. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 37
  11. 2025RePO: Replay-Enhanced Policy Optimizationcited by 36
  12. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  13. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  14. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoningcited by 35
  15. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 35
  16. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 34
  17. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  18. 2025GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learningcited by 33
  19. 2025NGRPO: Negative-enhanced Group Relative Policy Optimizationcited by 31
  20. 2025On-Policy RL with Optimal Reward Baselinecited by 30
  21. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  22. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  23. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  24. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 28
  25. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  26. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  27. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 27
  28. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 25
  29. 2025Rethinking Entropy Regularization in Large Reasoning Modelscited by 25
  30. 2025Nudging the Boundaries of LLM Reasoningcited by 25
  31. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 24
  32. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  33. 2025InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scalingcited by 19
  34. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 19
  35. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  36. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
  37. 2025CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Preventioncited by 17
  38. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17

50 most cited of 266 All 266 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think
  2. 2026LLMs Can Learn to Reason Via Off-Policy RL
  3. 2026Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
  4. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  5. 2026Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMs
  6. 2025Boosting Reasoning in Large Multimodal Models via Activation Replay
  7. 2025Learning to Reason Efficiently with Discounted Reinforcement Learning
  8. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners