mlsanityData to 7 Oct 2026

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

arXiv 2402.05808 ↗First posted 8 Feb 2024cs.AI cs.CL cs.LGcited by 63

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In this paper, we propose R3: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models. The core challenge in applying RL to complex reasoning is to identify a sequence of actions that result in positive rewards and provide appropriate supervision for optimization. Outcome supervision provides sparse rewards for final results without identifying error locations, whereas process supervision offers step-wise rewards but requires extensive manual annotation. R3 overcomes these limitations by learning from correct demonstrations. Specifically, R3 progressively slides the start state of reasoning from a demonstration's end to its beginning, facilitating easier model exploration at all stages. Thus, R3 establishes a step-wise curriculum, allowing outcome supervision to offer step-level signals and precisely pinpoint errors. Using Llama2-7B, our method surpasses RL baseline on eight reasoning tasks by 4.1 points on average. Notebaly, in program-based reasoning on GSM8K, it exceeds the baseline by 4.2 points across three backbone models, and without any extra data, Codellama-7B + R3 performs comparable to larger models or closed-source models.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 70 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

55 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Let's Verify Step by Step31 May
  2. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 52

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  6. 2017A Survey on Multi-Task Learningcited by 838
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  8. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  9. 2022Galactica: A Large Language Model for Sciencecited by 736
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  13. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  14. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  15. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  16. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  17. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  18. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  19. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  20. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  21. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  22. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  23. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  24. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  25. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  26. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  27. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  28. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  29. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  30. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  31. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
  32. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  33. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  34. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  35. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  36. 2023Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinementcited by 43
  37. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  38. 2016Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replaycited by 22

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 33; 2026: 19.

63 papers.

Sort:

By area:

  1. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  4. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  6. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  7. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  8. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  9. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  10. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
  11. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 58
  12. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
38 more
  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  2. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  3. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-trainingcited by 45
  4. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 39
  5. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
  6. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 35
  7. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  8. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Modelscited by 28
  9. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22
  10. 2024Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Modelscited by 20
  11. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  12. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 19
  13. 2024Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoningcited by 15
  14. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 15
  15. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  16. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13
  17. 2024Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewardscited by 11
  18. 2025IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmarkcited by 11
  19. 2025FreePRM: Training Process Reward Models Without Ground Truth Process Labelscited by 10
  20. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 10
  21. 2025From Reasoning to Super-Intelligence: A Search-Theoretic Perspectivecited by 9
  22. 2026Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Groundingcited by 9
  23. 2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
  24. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
  25. 2026PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learningcited by 7
  26. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 5
  27. 2025Improving Retrospective Language Agents via Joint Policy Gradient Optimizationcited by 4
  28. 2026A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curriculacited by 3
  29. 2026Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problemscited by 3
  30. 2025Trust-Region Adaptive Policy Optimizationcited by 2
  31. 2026Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoningcited by 2
  32. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  33. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  34. 2025SOLAR: Scalable Optimization of Large-scale Architecture for Reasoningcited by 1
  35. 2025From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communicationscited by 1
  36. 2025Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflectioncited by 1
  37. 2026TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Modelscited by 1
  38. 2025Improving Value-based Process Verifier via Structural Prior Injectioncited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
  2. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model
  3. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  4. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
  5. 2026TACLer: Tailored Curriculum Reinforcement Learning for Efficient Reasoning
  6. 2025Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math
  7. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  8. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning