# Nudging the Boundaries of LLM Reasoning

arXiv: https://arxiv.org/abs/2509.25666 · HTML: https://mlsanity.com/paper/2509.25666/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 44 of its 64 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 44)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Spurious Rewards: Rethinking Training Signals in RLVR](https://mlsanity.com/paper/2506.10947/) (2025)
- [ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models](https://mlsanity.com/paper/2505.24864/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [Llama-Nemotron: Efficient Reasoning Models](https://mlsanity.com/paper/2505.00949/) (2025)
- [Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining](https://mlsanity.com/paper/2504.07912/) (2025)
- [OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling](https://mlsanity.com/paper/2506.20512/) (2025)
- [Maximizing Confidence Alone Improves Reasoning](https://mlsanity.com/paper/2505.22660/) (2025)
- [Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions](https://mlsanity.com/paper/2506.07527/) (2025)

## Built on by (most cited first, 26 of 26)

- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration](https://mlsanity.com/paper/2601.18779/) (2026)
- [Maximum Likelihood Reinforcement Learning](https://mlsanity.com/paper/2602.02710/) (2026)
- [Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes](https://mlsanity.com/paper/2601.18795/) (2026)
- [Learn Hard Problems During RL with Reference Guided Fine-tuning](https://mlsanity.com/paper/2603.01223/) (2026)
- [Learning to Hint for Reinforcement Learning](https://mlsanity.com/paper/2604.00698/) (2026)
- [Expanding LLM Agent Boundaries with Strategy-Guided Exploration](https://mlsanity.com/paper/2603.02045/) (2026)
- [AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals](https://mlsanity.com/paper/2605.20643/) (2026)
- [Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models](https://mlsanity.com/paper/2602.12036/) (2026)
- [Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems](https://mlsanity.com/paper/2604.04767/) (2026)
- [Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR](https://mlsanity.com/paper/2605.15726/) (2026)
- [Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error](https://mlsanity.com/paper/2510.26109/) (2025)
- [Better, Faster: Harnessing Self-Improvement in Large Reasoning Models](https://mlsanity.com/paper/2605.24998/) (2026)
- [From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning](https://mlsanity.com/paper/2605.22074/) (2026)
- [Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematics](https://mlsanity.com/paper/2608.01522/) (2026)
- [Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy](https://mlsanity.com/paper/2608.02087/) (2026)
- [Making Expert Reasoning Learnable with Self-Distillation](https://mlsanity.com/paper/2602.02405/) (2026)
- [Selective Off-Policy Reference Tuning with Plan Guidance](https://mlsanity.com/paper/2605.11505/) (2026)
- [RLTL;DR: Self-improvement by Internalizing Self-generated Feedback](https://mlsanity.com/paper/2609.37633/) (2026)
- [AIPO: Learning to Reason from Active Interaction](https://mlsanity.com/paper/2605.08401/) (2026)
- [TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decomposition](https://mlsanity.com/paper/2606.09883/) (2026)
- [LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts](https://mlsanity.com/paper/2607.27787/) (2026)
- [Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching](https://mlsanity.com/paper/2609.07303/) (2026)
- [Teach to Learn: Hint Annealing for Self-improving LLM Reasoning](https://mlsanity.com/paper/2609.34975/) (2026)
- [Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR](https://mlsanity.com/paper/2609.37825/) (2026)
- [Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation](https://mlsanity.com/paper/2610.02700/) (2026)

## Abstract (arXiv)

Current online reinforcement learning (RL) algorithms like GRPO share a key limitation in LLM reasoning: they cannot learn from problems that are "unsolvable" to the model. In other words, they can only improve performance on problems where the model is capable of exploring the correct answer. Consequently, the model's "upper limit" remains unchanged after RL training, even though the likelihood of solving easier, solvable problems may increase. These hard samples cannot contribute to training, as no rollouts yield rewards and thus no gradients are produced. To unlock learning from these hard samples, we propose NuRL, a "nudging" method that aims to push the upper bound of LLM reasoning using self-generated hints, i.e., abstract cues that help reduce the problem difficulty for the model. Given a question and its gold answer, the model generates a CoT and then produces a hint containing the core knowledge needed to solve the problem. During training, we generate G rollouts from the base policy and use the pass rate to decide whether the hint should be injected. For hard samples with a 0% pass rate, we inject the hint and regenerate a new batch of trajectories. This yields two benefits: (1) the hint boosts pass rates (from 0% to non-zero), thereby introducing training signals for previously unsolvable samples, and (2) the hints are self-generated, avoiding distributional shift and do not rely on external models. NuRL achieves consistent improvements across 6 benchmarks and 3 models, while remaining complementary to test-time scaling. Notably, NuRL can raise the model's upper limit, whereas GRPO leaves pass@1024 unchanged from the base model. Furthermore, we present a systematic study of what makes an effective hint and when hints are most useful. Interestingly, the best hints are abstract and high-level, and are most beneficial when applied necessarily and after GRPO has converged.