mlsanityData to 14 Oct 2026

Failure Modes of Maximum Entropy RLHF

arXiv 2509.20265 ↗First posted 24 Sep 2025cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this paper, we show that Simple Preference Optimization (SimPO) can be derived as Maximum Entropy Reinforcement Learning, providing a theoretical foundation for this reference-free method. Motivated by SimPO's strong performance in offline preference optimization, we investigate whether Maximum Entropy RL can achieve similar results in online RLHF settings. Our experiments find that Maximum Entropy RL frequently exhibits overoptimization and unstable KL dynamics across model scales, with overoptimization persisting even at conservative learning rates for some configurations. Unlike KL-constrained methods that maintain stable training, entropy regularization fails to reliably prevent reward hacking and, in our experiments, correlates with the onset of overoptimization rather than guarding against it. Even in configurations where training remains stable, entropy regularization is not the stabilizing factor. Lastly, we discuss possible explanations for why SimPO succeeds in offline settings while Maximum Entropy RL struggles in online scenarios. Our findings suggest that reference-free approaches may face distinct challenges when applied to online versus offline preference learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 68 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

44 of 68 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  2. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  3. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  8. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
23 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 328
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  3. 2017Inverse Reward Designcited by 249
  4. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  6. 2018Understanding the impact of entropy on policy optimizationcited by 176
  7. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  8. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  9. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  10. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  11. 2019Truly Proximal Policy Optimizationcited by 69
  12. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  13. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  14. 2024Understanding Reference Policies in Direct Preference Optimizationcited by 22
  15. 2022A2C is a special case of PPOcited by 16
  16. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14
  17. 2025Robust Preference Optimization via Dynamic Target Marginscited by 13
  18. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
  19. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
  20. 2024A Practical Analysis of Human Alignment with *POcited by 7
  21. 2025When Maximum Entropy Misleads Policy Optimizationcited by 7
  22. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  23. 2022Maximum entropy exploration in contextual bandits with neural networks and energy based modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
  3. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  4. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation
  5. 2024Dataset Reset Policy Optimization for RLHF
  6. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  7. 2024Entropy Controllable Direct Preference Optimization
  8. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF