mlsanityData to 14 Oct 2026

Understanding Reinforcement Learning for Model Training, and future directions with GRAPE

arXiv 2509.04501 ↗First posted 2 Sep 2025cs.CL cs.AI cs.LGcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

This paper provides a self-contained, from-scratch, exposition of key algorithms for instruction tuning of models: SFT, Rejection Sampling, REINFORCE, Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Explanations of these algorithms often assume prior knowledge, lack critical details, and/or are overly generalized and complex. Here, each method is discussed and developed step by step using simplified and explicit notation focused on LLMs, aiming to eliminate ambiguity and provide a clear and intuitive understanding of the concepts. By minimizing detours into the broader RL literature and connecting concepts to LLMs, we eliminate superfluous abstractions and reduce cognitive overhead. Following this exposition, we provide a literature review of new techniques and approaches beyond those detailed. Finally, new ideas for research and exploration in the form of GRAPE (Generalized Relative Advantage Policy Evolution) are presented.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 73 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

38 of 73 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning7 Jun
  2. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression27 May
  3. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoning20 Dec
  4. 2024PSPO*: An Effective Process-supervised Policy Optimization for Reasoning Alignment18 Nov
  5. 2023Let's Verify Step by Step31 May
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  8. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
17 more
  1. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  2. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  3. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  4. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  5. 2021Curriculum Learning: A Surveycited by 259
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  7. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  8. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  9. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  10. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 41
  11. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  12. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 32
  13. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  14. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  15. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedbackcited by 15
  16. 2025C3AI: Crafting and Evaluating Constitutions for Constitutional AIcited by 13
  17. 2025Process-Supervised LLM Recommenders via Flow-guided Tuningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026GIM: Evaluating models via tasks that integrate multiple cognitive domainscited by 1
  2. 2026Training-Inference Kernel Contracts: Bounding Divergence in Post-Training and Deploymentcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
  2. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  3. 2025A Practical Introduction to Deep Reinforcement Learning
  4. 2026Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
  5. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  6. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  7. 2026GOPO: Policy Optimization using Ranked Rewards
  8. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning