mlsanityData to 14 Oct 2026

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

arXiv 2310.10505 ↗First posted 16 Oct 2023cs.LGcited by 187

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is key to aligning Large Language Models (LLMs), typically paired with the Proximal Policy Optimization (PPO) algorithm. While PPO is a powerful method designed for general reinforcement learning tasks, it is overly sophisticated for LLMs, leading to laborious hyper-parameter tuning and significant computation burdens. To make RLHF efficient, we present ReMax, which leverages 3 properties of RLHF: fast simulation, deterministic transitions, and trajectory-level rewards. These properties are not exploited in PPO, making it less suitable for RLHF. Building on the renowned REINFORCE algorithm, ReMax does not require training an additional value model as in PPO and is further enhanced with a new variance reduction technique. ReMax offers several benefits over PPO: it is simpler to implement, eliminates more than 4 hyper-parameters in PPO, reduces GPU memory usage, and shortens training time. ReMax can save about 46% GPU memory than PPO when training a 7B model and enables training on A800-80GB GPUs without the memory-saving offloading technique needed by PPO. Applying ReMax to a Mistral-7B model resulted in a 94.78% win rate on the AlpacaEval leaderboard and a 7.739 score on MT-bench, setting a new SOTA for open-source 7B models. These results show the effectiveness of ReMax while addressing the limitations of PPO in LLMs.

Key facts. Cited by 187 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 71 of its 104 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

71 of 104 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  3. 2023Efficient RLHF: Reducing the Memory Usage of PPO1 Sep
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  5. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  7. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023Reward Collapse in Aligning Large Language Models28 May
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
3 more
  1. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2023Mistral 7Bcited by 3,164
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  13. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  14. 2023LIMA: Less Is More for Alignmentcited by 1,100
  15. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  16. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  17. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  18. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  19. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  20. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  21. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  22. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  23. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  24. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  25. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  26. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  27. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  28. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  29. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  30. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  31. 2015Minimum Risk Training for Neural Machine Translationcited by 115
  32. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  33. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  34. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  35. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85
  36. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 72
  37. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 60
  38. 2019Low-Memory Neural Network Training: A Technical Reportcited by 57

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 26; 2025: 91; 2026: 68.

187 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  5. 2025Learning to Reason under Off-Policy Guidancecited by 281
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  7. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  9. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 150
  10. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  11. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  12. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
38 more
  1. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  2. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  3. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 108
  4. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  5. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  6. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  7. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  8. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 65
  9. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 61
  10. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  11. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  12. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  13. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  14. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 47
  15. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  16. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  17. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  18. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  19. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  20. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  21. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 37
  22. 2025A Survey of Direct Preference Optimizationcited by 36
  23. 2025On-Policy RL with Optimal Reward Baselinecited by 34
  24. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  25. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  26. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 29
  27. 2025BNPO: Beta Normalization Policy Optimizationcited by 28
  28. 2025Optimizing Anytime Reasoning via Budget Relative Policy Optimizationcited by 27
  29. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimationcited by 27
  30. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  31. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 26
  32. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  33. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  34. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  35. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  36. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 23
  37. 2023Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuningcited by 22
  38. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerated Preference Optimization for Large Language Model Alignment
  2. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  3. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  4. 2026Efficient Hyperparameter Optimization for LLM Reinforcement Learning
  5. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
  6. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  7. 2025A Principled Loss Function for Direct Language Model Alignment
  8. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions