mlsanityData to 14 Oct 2026

Reinforcement Learning for Reasoning in Large Language Models with One Training Example

arXiv 2504.20571 ↗First posted 29 Apr 2025cs.LG cs.AI cs.CLcited by 295

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0% to 73.6% (8.6% improvement beyond format correction), and improves the average performance across six common mathematical reasoning benchmarks from 17.6% to 35.7% (7.0% non-format gain). This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6%, average: 35.9%), which contains the aforementioned example. Furthermore, RLVR with only two examples even slightly exceeds these results (MATH500: 74.8%, average: 36.6%). Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples. In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post-saturation generalization. Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., by incorporating entropy loss with an appropriate coefficient) in 1-shot RLVR training. We also further discuss related observations about format correction, label robustness and prompt modification. These findings can inspire future work on RLVR efficiency and encourage a re-examination of recent progress and the underlying mechanisms in RLVR. All resources are open source at https://github.com/ypwang61/One-Shot-RLVR.

Key facts. Cited by 295 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 68 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

52 of 68 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  2. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  5. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  10. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  11. 2025LIMR: Less is More for RL Scaling17 Feb
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
4 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  2. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
24 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  8. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  9. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  10. 2017Deep Learning is Robust to Massive Label Noisecited by 165
  11. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  12. 2025Concise Reasoning via Reinforcement Learningcited by 79
  13. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
  14. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  15. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  16. 2024Active Preference Learning for Large Language Modelscited by 58
  17. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  18. 2025Rethinking Reflection in Pre-Trainingcited by 43
  19. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  20. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 27
  21. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  22. 2024The Complexity Dynamics of Grokkingcited by 22
  23. 2024Deep Grokking: Would Deep Neural Networks Generalize Better?cited by 15
  24. 2024Enabling Weak LLMs to Judge Response Reliability via Meta Rankingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 162; 2026: 133.

295 papers.

Sort:

By area:

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  3. 2025Memory in the Age of AI Agentscited by 303
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  5. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  7. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  8. 2025Generalizing Verifiable Instruction Followingcited by 194
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  10. 2025Hierarchical Reasoning Modelcited by 177
  11. 2025Agentic Reinforced Policy Optimizationcited by 172
  12. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
38 more
  1. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  2. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  3. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 113
  4. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  5. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  6. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  7. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 82
  8. 2025Kevin: Multi-Turn RL for Generating CUDA Kernelscited by 78
  9. 2025Can Large Reasoning Models Self-Train?cited by 68
  10. 2025ThetaEvolve: Test-time Learning on Open Problemscited by 64
  11. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  12. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  13. 2025Training a Scientific Reasoning Model for Chemistrycited by 56
  14. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  15. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
  16. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  17. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  18. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  19. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  20. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  21. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
  22. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 42
  23. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  24. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  25. 2025Reinforcement Pre-Trainingcited by 40
  26. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  27. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  28. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  29. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 38
  30. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 35
  31. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  32. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 33
  33. 2025One-shot Entropy Minimizationcited by 32
  34. 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learningcited by 31
  35. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 29
  36. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 29
  37. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoningcited by 28
  38. 2026Towards Execution-Grounded Automated AI Researchcited by 28

50 most cited of 295 All 295 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  2. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  4. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  5. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
  6. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  7. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning
  8. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning