mlsanityData to 7 Oct 2026

Teaching Large Language Models to Reason with Reinforcement Learning

arXiv 2403.04642 ↗First posted 7 Mar 2024cs.LGcited by 158

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (PPO), Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate both sparse and dense rewards provided to the LLM both heuristically and via a learned reward model. We additionally start from multiple model sizes and initializations both with and without supervised fine-tuning (SFT) data. Overall, we find all algorithms perform comparably, with Expert Iteration performing best in most cases. Surprisingly, we find the sample complexity of Expert Iteration is similar to that of PPO, requiring at most on the order of 106 samples to converge from a pretrained checkpoint. We investigate why this is the case, concluding that during RL training models fail to explore significantly beyond solutions already produced by SFT models. Additionally, we discuss a trade off between maj@1 and pass@96 metric performance during SFT training and how conversely RL training improves both simultaneously. We then conclude by discussing the implications of our findings for RLHF and the future role of RL in LLM fine-tuning.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 65 of its 84 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

65 of 84 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Let's Verify Step by Step31 May
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  6. 2017Proximal Policy Optimization Algorithms20 Jul
  7. 2017Thinking Fast and Slow with Deep Learning and Tree Search23 May

Other references · 58

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  13. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  14. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  15. 2023StarCoder: may the source be with you!cited by 790
  16. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  17. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  18. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  20. 2022Large Language Models Can Self-Improvecited by 656
  21. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  22. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  23. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  24. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  25. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  26. 2023Llemma: An Open Language Model For Mathematicscited by 385
  27. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  28. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  29. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  30. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  31. 2020Prioritized Level Replaycited by 190
  32. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  33. 2022Evolution through Large Modelscited by 131
  34. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  35. 2022Language Model Cascadescited by 104
  36. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  37. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 76
  38. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 46; 2025: 82; 2026: 30.

158 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  9. 2024Iterative Reasoning Preference Optimizationcited by 244
  10. 2024TradingAgents: Multi-Agents LLM Financial Trading Frameworkcited by 182
  11. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  12. 2024The pitfalls of next-token predictioncited by 156
38 more
  1. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  2. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  3. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 109
  4. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  5. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  6. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  7. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  8. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  9. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  10. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  11. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  12. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  13. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 70
  14. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  15. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  16. 2025Training a Scientific Reasoning Model for Chemistrycited by 54
  17. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  18. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  19. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  20. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  21. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  22. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  23. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42
  24. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  25. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  26. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Modelscited by 36
  27. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  28. 2025RePO: Replay-Enhanced Policy Optimizationcited by 36
  29. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimizationcited by 34
  30. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  31. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  32. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  33. 2025NGRPO: Negative-enhanced Group Relative Policy Optimizationcited by 31
  34. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  35. 2025Training a Generally Curious Agentcited by 29
  36. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  37. 2025FlashThink: An Early Exit Method For Efficient Reasoningcited by 28
  38. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Criticcited by 27

50 most cited of 158 All 158 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
  2. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
  3. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  4. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  5. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  6. 2026ExpRL: Exploratory RL for LLM Mid-Training
  7. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  8. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning