mlsanityData to 14 Oct 2026

Reinforcement Learning for LLM Post-Training: A Survey

arXiv 2407.16216 ↗First posted 23 Jul 2024cs.CLcited by 92

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large language models (LLMs) trained via pretraining and supervised fine-tuning (SFT) can still produce harmful and misaligned outputs, or struggle in domains like math and coding. Reinforcement learning (RL)-based post-training methods, including Reinforcement Learning from Human Feedback (RLHF) methods like Direct Preference Optimization (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR) approaches like PPO and GRPO, have made remarkable gains to alleviate these issues. Yet, no existing work offers a technically detailed comparison of the various methods driving this progress. In order to fill this gap, we present a timely survey that connects foundational components with latest advancements. We derive a single policy gradient framework that unifies pretraining, SFT, RLHF, and RLVR as special cases while also organizing the more recent techniques therein. The main contributions of our survey are as follows: (1) a self-contained introduction to MLE, RLHF, and RLVR foundations and the unified policy gradient framework; (2) detailed technical analysis of PPO- and GRPO-based methods alongside offline and iterative DPO approaches, decomposed along prompt sampling, response sampling, and gradient coefficient axes; (3) standardized notation enabling direct cross-method comparison; and (4) comprehensive comparison of implementation details and empirical results of each method in the appendix. We aim to serve as a technically grounded reference for researchers and practitioners working on LLM post-training.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. First posted on arXiv 23 Jul 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 62; 2026: 20.

92 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  4. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 63
  5. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  6. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 61
  7. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  8. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  9. 2025A Survey of Direct Preference Optimizationcited by 36
  10. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  11. 2025JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restorationcited by 34
  12. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
38 more
  1. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  2. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  3. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  4. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 16
  5. 2025ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learningcited by 16
  6. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  7. 2024Superficial Safety Alignment Hypothesiscited by 14
  8. 2025Following the Autoregressive Nature of LLM Embeddings via Compression and Alignmentcited by 14
  9. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 14
  10. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  11. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 13
  12. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  13. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  14. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?cited by 12
  15. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10
  16. 2025Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perceptioncited by 10
  17. 2025Self-Destructive Language Modelcited by 10
  18. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 10
  19. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signalscited by 9
  20. 2025Brevity is the soul of sustainability: Characterizing LLM response lengthscited by 9
  21. 2025In-Distribution Steering: Balancing Control and Coherence in Language Model Generationcited by 9
  22. 2025Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluationcited by 9
  23. 2025Graph-Augmented Reasoning: Evolving Step-by-Step Knowledge Graph Retrieval for LLM Reasoningcited by 8
  24. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  25. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  26. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7
  27. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generationcited by 7
  28. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrievalcited by 7
  29. 2025Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Modelcited by 7
  30. 2025Clinical-R1: Empowering Large Language Models for Faithful and Comprehensive Reasoning with Clinical Objective Relative Policy Optimizationcited by 7
  31. 2024SPICA: Retrieving Scenarios for Pluralistic In-Context Alignmentcited by 6
  32. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 6
  33. 2024COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Frameworkcited by 5
  34. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolationcited by 5
  35. 2025Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solvingcited by 4
  36. 2025Domain-Specific Constitutional AI: Enhancing Safety in LLM-Powered Mental Health Chatbotscited by 4
  37. 2025Reward Model Routing in Alignmentcited by 4
  38. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  2. 2024Reinforcement Learning Enhanced LLMs: A Survey
  3. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2026Demystifying Reinforcement Learning Post-Training of Language Models
  6. 2025RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs
  7. 2025Towards a Unified View of Large Language Model Post-Training
  8. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model