mlsanityData to 14 Oct 2026

POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration

arXiv 2601.18779 ↗First posted 26 Jan 2026cs.LG cs.AI cs.CLcited by 42

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has improved the reasoning abilities of large language models (LLMs), yet state-of-the-art methods still fail to learn on many training problems. On hard problems, on-policy RL rarely explores even a single correct rollout, yielding zero reward and no learning signal for driving improvement. We find that natural solutions to remedy this exploration problem from classical RL, such as entropy bonuses, more permissive clipping of the importance ratio, or direct optimization of pass@k objectives, do not resolve this issue and often destabilize optimization without improving solvability. A natural alternative is to leverage transfer from easier problems. However, we show that mixing easy and hard problems during RL training is counterproductive due to ray interference, where optimization focuses on already-solvable problems in a way that actively inhibits progress on harder ones. To address this challenge, we introduce Privileged On-Policy Exploration (POPE), an approach that leverages human- or other oracle solutions as privileged information to guide exploration on hard problems, unlike methods that use oracle solutions as training targets (e.g., off-policy RL methods or warmstarting from SFT). POPE augments hard problems with prefixes of oracle solutions, enabling RL to obtain non-zero rewards during guided rollouts. Crucially, the resulting behaviors transfer back to the original, unguided problems through a synergy between instruction-following and reasoning. Empirically, POPE expands the set of solvable problems and substantially improves performance on challenging reasoning benchmarks.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 54 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

40 of 54 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning20 Jan
  2. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration1 Oct
  3. 2025Nudging the Boundaries of LLM Reasoning30 Sep
  4. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation23 Sep
  5. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  6. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation17 Jul
  7. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  8. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  9. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  10. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  11. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
  12. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
8 more
  1. 2025Learning to Reason under Off-Policy Guidance21 Apr
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  4. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  5. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  6. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  8. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  9. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  10. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  11. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  12. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
8 more
  1. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  2. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  3. 2024InfAlign: Inference-aware language model alignmentcited by 35
  4. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
  5. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 16
  6. 2025Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RLcited by 11
  7. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
  8. 2025Polychromic Objectives for Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 42.

42 papers.

Sort:

By area:

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  2. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 39
  3. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
  4. 2026Scaling Self-Play with Self-Guidancecited by 10
  5. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 10
  6. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 8
  7. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  8. 2026Learning from Language Feedback via Variational Policy Distillationcited by 7
  9. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  10. 2026SE-Bench: Benchmarking Self-Evolution with Knowledge Internalizationcited by 6
  11. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RLcited by 6
  12. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVRcited by 4
30 more
  1. 2026Agent Explorative Policy Optimization for Multimodal Agentic Reasoningcited by 4
  2. 2026Scaling Automatic Research Agents via World Modelscited by 4
  3. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioningcited by 3
  4. 2026Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillationcited by 3
  5. 2026Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problemscited by 3
  6. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  7. 2026Tail-Likelihood Reinforcement Learningcited by 3
  8. 2026From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoningcited by 2
  9. 2026Self-Trained Verification for Training- and Test-Time Self-Improvementcited by 2
  10. 2026AutoOR: Scalably Post-training LLMs to Autoformulate Operations Research Problemscited by 1
  11. 2026Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regimecited by 1
  12. 2026SCOPE-RL: Optimizing Reasoning Paths Before and After Successcited by 1
  13. 2026MIRROR: Learning from the Other View for Multi-Modal Reasoningcited by 1
  14. 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoningcited by 1
  15. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
  16. 2026PolicyLong: Towards On-Policy Context Extensioncited by 0
  17. 2026Hide to Guide: Learning via Semantic Maskingcited by 0
  18. 2026DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixescited by 0
  19. 2026ExpRL: Exploratory RL for LLM Mid-Trainingcited by 0
  20. 2026REVES: REvision and VErification--Augmented Training for Test-Time Scalingcited by 0
  21. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  22. 2026ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalizationcited by 0
  23. 2026Long-Horizon Language Model Reinforcement Learning via Progressive Point Matchingcited by 0
  24. 2026From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agentscited by 0
  25. 2026Teach to Learn: Hint Annealing for Self-improving LLM Reasoningcited by 0
  26. 2026Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVRcited by 0
  27. 2026Finetuning with Sampling: SFT Learns Better Than You Thinkcited by 0
  28. 2026Balancing Reference Guidance and Free Generation in Trajectory Rollouts for Reasoning RLcited by 0
  29. 2026MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesiscited by 0
  30. 2026Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
  2. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches
  3. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  4. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
  5. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  6. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  7. 2025Reasoning with Exploration: An Entropy Perspective
  8. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation