mlsanityData to 14 Oct 2026

Nudging the Boundaries of LLM Reasoning

arXiv 2509.25666 ↗First posted 30 Sep 2025cs.LG cs.CLcited by 26

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Current online reinforcement learning (RL) algorithms like GRPO share a key limitation in LLM reasoning: they cannot learn from problems that are "unsolvable" to the model. In other words, they can only improve performance on problems where the model is capable of exploring the correct answer. Consequently, the model's "upper limit" remains unchanged after RL training, even though the likelihood of solving easier, solvable problems may increase. These hard samples cannot contribute to training, as no rollouts yield rewards and thus no gradients are produced. To unlock learning from these hard samples, we propose NuRL, a "nudging" method that aims to push the upper bound of LLM reasoning using self-generated hints, i.e., abstract cues that help reduce the problem difficulty for the model. Given a question and its gold answer, the model generates a CoT and then produces a hint containing the core knowledge needed to solve the problem. During training, we generate G rollouts from the base policy and use the pass rate to decide whether the hint should be injected. For hard samples with a 0% pass rate, we inject the hint and regenerate a new batch of trajectories. This yields two benefits: (1) the hint boosts pass rates (from 0% to non-zero), thereby introducing training signals for previously unsolvable samples, and (2) the hints are self-generated, avoiding distributional shift and do not rely on external models. NuRL achieves consistent improvements across 6 benchmarks and 3 models, while remaining complementary to test-time scaling. Notably, NuRL can raise the model's upper limit, whereas GRPO leaves pass@1024 unchanged from the base model. Furthermore, we present a systematic study of what makes an effective hint and when hints are most useful. Interestingly, the best hints are abstract and high-level, and are most beneficial when applied necessarily and after GRPO has converged.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 64 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

44 of 64 references.

Closest ancestors · 31

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025MiGrATe: Mixed-Policy GRPO for Adaptation at Test-Time12 Aug
  3. 2025A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning18 Jul
  4. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation17 Jul
  5. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason3 Jul
  6. 2025ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning3 Jul
  7. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling25 Jun
  8. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  9. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  10. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  11. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  12. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO9 Jun
19 more
  1. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning5 Jun
  2. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback3 Jun
  3. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  5. 2025Maximizing Confidence Alone Improves Reasoning28 May
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  8. 2025Learning to Reason under Off-Policy Guidance21 Apr
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  11. 2025Boosting MLLM Reasoning with Text-Debiased Hint-GRPO31 Mar
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  13. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  14. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  15. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  16. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  17. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  18. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  19. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  10. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  11. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  12. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
1 more
  1. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 25.

26 papers.

Sort:

By area:

  1. 2026Privileged Information Distillation for Language Modelscited by 90
  2. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  3. 2026Maximum Likelihood Reinforcement Learningcited by 36
  4. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  5. 2026Learn Hard Problems During RL with Reference Guided Fine-tuningcited by 12
  6. 2026Learning to Hint for Reinforcement Learningcited by 12
  7. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 9
  8. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 7
  9. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  10. 2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
  11. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVRcited by 4
  12. 2025Do Not Step Into the Same River Twice: Learning to Reason from Trial and Errorcited by 3
14 more
  1. 2026Better, Faster: Harnessing Self-Improvement in Large Reasoning Modelscited by 3
  2. 2026From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoningcited by 2
  3. 2026Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematicscited by 2
  4. 2026Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policycited by 2
  5. 2026Making Expert Reasoning Learnable with Self-Distillationcited by 1
  6. 2026Selective Off-Policy Reference Tuning with Plan Guidancecited by 1
  7. 2026RLTL;DR: Self-improvement by Internalizing Self-generated Feedbackcited by 1
  8. 2026AIPO: Learning to Reason from Active Interactioncited by 0
  9. 2026TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decompositioncited by 0
  10. 2026LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Promptscited by 0
  11. 2026Long-Horizon Language Model Reinforcement Learning via Progressive Point Matchingcited by 0
  12. 2026Teach to Learn: Hint Annealing for Self-improving LLM Reasoningcited by 0
  13. 2026Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVRcited by 0
  14. 2026Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  2. 2026The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
  3. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
  4. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance
  5. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  6. 2025MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs
  7. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't
  8. 2025Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings