# Reinforcement Learning for LLM Post-Training: A Survey

arXiv: https://arxiv.org/abs/2407.16216 · HTML: https://mlsanity.com/paper/2407.16216/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- First posted on arXiv 23 Jul 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Built on by (most cited first, 30 of 92)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Aligning LLMs with Individual Preferences via Interaction](https://mlsanity.com/paper/2410.03642/) (2024)
- [A Systematic Survey of Automatic Prompt Optimization Techniques](https://mlsanity.com/paper/2502.16923/) (2025)
- [RMB: Comprehensively Benchmarking Reward Models in LLM Alignment](https://mlsanity.com/paper/2410.09893/) (2024)
- [Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond](https://mlsanity.com/paper/2502.19301/) (2025)
- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment](https://mlsanity.com/paper/2408.06266/) (2024)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle](https://mlsanity.com/paper/2509.16679/) (2025)
- [JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration](https://mlsanity.com/paper/2504.04158/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuning](https://mlsanity.com/paper/2503.11441/) (2025)
- [RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization](https://mlsanity.com/paper/2410.04203/) (2024)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [Advancing LLM Safe Alignment with Safety Representation Ranking](https://mlsanity.com/paper/2505.15710/) (2025)
- [ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learning](https://mlsanity.com/paper/2508.17608/) (2025)
- [A Survey on Progress in LLM Alignment from the Perspective of Reward Design](https://mlsanity.com/paper/2505.02666/) (2025)
- [Superficial Safety Alignment Hypothesis](https://mlsanity.com/paper/2410.10862/) (2024)
- [Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment](https://mlsanity.com/paper/2502.11401/) (2025)
- [EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework](https://mlsanity.com/paper/2506.22200/) (2025)
- [Unveiling the Basin-Like Loss Landscape in Large Language Models](https://mlsanity.com/paper/2505.17646/) (2025)
- [An Embarrassingly Simple Defense Against LLM Abliteration Attacks](https://mlsanity.com/paper/2505.19056/) (2025)
- [Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences](https://mlsanity.com/paper/2506.02698/) (2025)
- [MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training](https://mlsanity.com/paper/2502.11541/) (2025)
- [Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?](https://mlsanity.com/paper/2502.19557/) (2025)
- [Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment](https://mlsanity.com/paper/2502.00203/) (2025)
- [Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception](https://mlsanity.com/paper/2504.20468/) (2025)
- [Self-Destructive Language Model](https://mlsanity.com/paper/2505.12186/) (2025)
- [Backdoor Attribution: Elucidating and Controlling Backdoor in Language Models](https://mlsanity.com/paper/2509.21761/) (2025)

## Abstract (arXiv)

Large language models (LLMs) trained via pretraining and supervised fine-tuning (SFT) can still produce harmful and misaligned outputs, or struggle in domains like math and coding. Reinforcement learning (RL)-based post-training methods, including Reinforcement Learning from Human Feedback (RLHF) methods like Direct Preference Optimization (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR) approaches like PPO and GRPO, have made remarkable gains to alleviate these issues. Yet, no existing work offers a technically detailed comparison of the various methods driving this progress. In order to fill this gap, we present a timely survey that connects foundational components with latest advancements. We derive a single policy gradient framework that unifies pretraining, SFT, RLHF, and RLVR as special cases while also organizing the more recent techniques therein. The main contributions of our survey are as follows: (1) a self-contained introduction to MLE, RLHF, and RLVR foundations and the unified policy gradient framework; (2) detailed technical analysis of PPO- and GRPO-based methods alongside offline and iterative DPO approaches, decomposed along prompt sampling, response sampling, and gradient coefficient axes; (3) standardized notation enabling direct cross-method comparison; and (4) comprehensive comparison of implementation details and empirical results of each method in the appendix. We aim to serve as a technically grounded reference for researchers and practitioners working on LLM post-training.