mlsanityData to 7 Oct 2026

Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't

arXiv 2503.16219 ↗First posted 20 Mar 2025cs.LG cs.CLcited by 72

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Enhancing the reasoning capabilities of large language models (LLMs) typically relies on massive computational resources and extensive datasets, limiting accessibility for resource-constrained settings. Our study investigates the potential of reinforcement learning (RL) to improve reasoning in small LLMs, focusing on a 1.5-billion-parameter model, DeepSeek-R1-Distill-Qwen-1.5B, under strict constraints: training on 4 NVIDIA A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy Optimization (GRPO) algorithm and curating a compact, high-quality mathematical reasoning dataset, we conducted three experiments to explore model behavior and performance. Our results demonstrate rapid reasoning gains - e.g., AMC23 accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing o1-preview - using only 7,000 samples and a 42 training cost, compared to thousands of dollars for baseline models. However, challenges such as optimization instability and length constraints emerged with prolonged training. These findings highlight the efficacy of RL-based fine-tuning for small LLMs, offering a cost-effective alternative to large-scale approaches. We release our code and datasets as open-source resources, providing insights into trade-offs and laying a foundation for scalable, reasoning-capable LLMs in resource-limited environments. All are available at https://github.com/knoveleng/open-rs.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 44 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

26 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Process Reinforcement through Implicit Rewards3 Feb
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2023Let's Verify Step by Step31 May
  4. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 22

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2025s1: Simple test-time scalingcited by 1,311
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
10 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  4. 2025LIMO: Less is More for Reasoningcited by 488
  5. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  6. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  7. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  8. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  9. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  10. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 43; 2026: 29.

72 papers.

Sort:

By area:

  1. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  2. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  3. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  4. 2025Phi-4-reasoning Technical Reportcited by 146
  5. 2025Efficient Reasoning Models: A Surveycited by 93
  6. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  7. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  8. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 59
  9. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  10. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39
  11. 2025Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQLcited by 38
  12. 2025Tina: Tiny Reasoning Models via LoRAcited by 23
38 more
  1. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  2. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  3. 2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19
  4. 2025DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoningcited by 18
  5. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 18
  6. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  7. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 14
  8. 2025RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguardscited by 14
  9. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  10. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 13
  11. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 11
  12. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Searchcited by 11
  13. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  14. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidancecited by 10
  15. 2025GRPO is Secretly a Process Reward Modelcited by 10
  16. 2025Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectorscited by 8
  17. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 8
  18. 2026Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learningcited by 7
  19. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
  20. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweightingcited by 6
  21. 2025Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?cited by 5
  22. 2025Resa: Transparent Reasoning Models via SAEscited by 4
  23. 2026BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoningcited by 4
  24. 2025AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margincited by 3
  25. 2025Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problemcited by 3
  26. 2025Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Designcited by 3
  27. 2025Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templatescited by 3
  28. 2025Reinforcement Learning for Latent-Space Thinking in LLMscited by 3
  29. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learningcited by 3
  30. 2025Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuningcited by 2
  31. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 2
  32. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 2
  33. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweightingcited by 2
  34. 2026RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimizationcited by 2
  35. 2026R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learningcited by 2
  36. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learningcited by 2
  37. 2025Effective Learning for Small Reasoning Models: An Empirical Study on 0.5B Reasoning LLMscited by 1
  38. 2025Replacing thinking with tool usage enables reasoning in small language modelscited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs
  3. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  5. 2025Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
  6. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  7. 2026Are complicated loss functions necessary for teaching LLMs to reason?
  8. 2025Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement