mlsanityData to 14 Oct 2026

A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce

arXiv 2504.11343 ↗First posted 15 Apr 2025cs.LG cs.AI cs.CLcited by 145

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has become a prevailing approach for fine-tuning large language models (LLMs) on complex reasoning tasks. Among recent methods, GRPO stands out for its empirical success in training models such as DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In this work, we revisit GRPO from a reinforce-like algorithm perspective and analyze its core components. Surprisingly, we find that a simple rejection sampling baseline, RAFT, which trains only on positively rewarded samples, yields competitive performance than GRPO and PPO. Our ablation studies reveal that GRPO's main advantage arises from discarding prompts with entirely incorrect responses, rather than from its reward normalization. Motivated by this insight, we propose Reinforce-Rej, a minimal extension of policy gradient that filters both entirely incorrect and entirely correct samples. Reinforce-Rej improves KL efficiency and stability, serving as a lightweight yet effective alternative to more complex RL algorithms. We advocate RAFT as a robust and interpretable baseline, and suggest that future advances should focus on more principled designs for incorporating negative samples, rather than relying on them indiscriminately. Our findings provide guidance for future work in reward-based LLM post-training.

Key facts. Cited by 145 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 41 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

34 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoning20 Dec
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  6. 2017Proximal Policy Optimization Algorithms20 Jul
  7. 2017Thinking Fast and Slow with Deep Learning and Tree Search23 May

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
15 more
  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  2. 2024Self-Rewarding Language Modelscited by 681
  3. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  7. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  8. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  9. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  10. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  11. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  12. 2025Self-rewarding correction for mathematical reasoningcited by 70
  13. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  14. 2024Automatic Curriculum Expert Iteration for Reliable LLM Reasoningcited by 16
  15. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 88; 2026: 57.

145 papers.

Sort:

By area:

  1. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  4. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 113
  5. 2025Geometric-Mean Policy Optimizationcited by 111
  6. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  7. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 89
  8. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  9. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  10. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  11. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
  12. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
38 more
  1. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  2. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  3. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
  4. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  5. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  6. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  7. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 44
  8. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  9. 2025The Hallucination Tax of Reinforcement Finetuningcited by 40
  10. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  11. 2026Maximum Likelihood Reinforcement Learningcited by 36
  12. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 34
  13. 2025NGRPO: Negative-enhanced Group Relative Policy Optimizationcited by 32
  14. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  15. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 31
  16. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  17. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 28
  18. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  19. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 24
  20. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  21. 2025GEM: A Gym for Agentic LLMscited by 23
  22. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  23. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 20
  24. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 20
  25. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 20
  26. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  27. 2025Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Modelscited by 18
  28. 2025What is the objective of reasoning with reinforcement learning?cited by 18
  29. 2025BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actionscited by 17
  30. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  31. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 15
  32. 2025ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learningcited by 13
  33. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  34. 2025ROC-n-reroll: How verifier imperfection affects test-time scalingcited by 12
  35. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  36. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Trainingcited by 12
  37. 2025Simple Policy Gradients for Reasoning with Diffusion Language Modelscited by 12
  38. 2026Future-KL Regularized GRPO: Process-Level Credit Assignment from f-Divergence Regularizationcited by 12

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  2. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  3. 2025Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement
  4. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
  5. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
  6. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  7. 2025Token-Efficient RL for LLM Reasoning
  8. 2026iGRPO: Self-Feedback-Driven LLM Reasoning