mlsanityData to 7 Oct 2026

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

arXiv 2402.14740 ↗First posted 22 Feb 2024cs.LGcited by 103

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. Proximal Policy Optimization (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF. However, it involves both high computational cost and sensitive hyperparameter tuning. We posit that most of the motivational principles that led to the development of PPO are less of a practical concern in RLHF and advocate for a less computationally expensive method that preserves and even increases performance. We revisit the formulation of alignment from human preferences in the context of RL. Keeping simplicity as a guiding principle, we show that many components of PPO are unnecessary in an RLHF context and that far simpler REINFORCE-style optimization variants outperform both PPO and newly proposed "RL-free" methods such as DPO and RAFT. Our work suggests that careful adaptation to LLMs alignment characteristics enables benefiting from online RL optimization at low cost.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 55 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

34 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  4. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2023PaLM 2 Technical Reportcited by 1,154
10 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  6. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  7. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  8. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  9. 2020A Distributional Approach to Controlled Text Generationcited by 117
  10. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 69; 2026: 27.

103 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  7. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  8. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  9. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  10. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  11. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  12. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
38 more
  1. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
  2. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 43
  3. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
  4. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  5. 2025Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agentscited by 37
  6. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 34
  7. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  8. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  9. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPOcited by 31
  10. 2025SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentscited by 29
  11. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  12. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  13. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  14. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 25
  15. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  16. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  17. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  18. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 23
  19. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 23
  20. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 21
  21. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21
  22. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  23. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 21
  24. 2025Boosting LLM Reasoning via Spontaneous Self-Correctioncited by 18
  25. 2024Automatic Curriculum Expert Iteration for Reliable LLM Reasoningcited by 16
  26. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  27. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  28. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
  29. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  30. 2025R2ec: Towards Large Recommender Models with Reasoningcited by 13
  31. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 13
  32. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  33. 2025ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Searchcited by 12
  34. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 12
  35. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  36. 2025Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoningcited by 11
  37. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
  38. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  4. 2024Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024Accelerated Preference Optimization for Large Language Model Alignment
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  8. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback