mlsanityData to 7 Oct 2026

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

arXiv 2501.19358 ↗First posted 31 Jan 2025cs.LGcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This work identifies the Energy Loss Phenomenon in Reinforcement Learning from Human Feedback (RLHF) and its connection to reward hacking. Specifically, energy loss in the final layer of a Large Language Model (LLM) gradually increases during the RL process, with an excessive increase in energy loss characterizing reward hacking. Beyond empirical analysis, we further provide a theoretical foundation by proving that, under mild conditions, the increased energy loss reduces the upper bound of contextual relevance in LLMs, which is a critical aspect of reward hacking as the reduced contextual relevance typically indicates overfitting to reward model-favored patterns in RL. To address this issue, we propose an Energy loss-aware PPO algorithm (EPPO) which penalizes the increase in energy loss in the LLM's final layer during reward calculation to prevent excessive energy loss, thereby mitigating reward hacking. We theoretically show that EPPO can be conceptually interpreted as an entropy-regularized RL algorithm, which provides deeper insights into its effectiveness. Extensive experiments across various LLMs and tasks demonstrate the commonality of the energy loss phenomenon, as well as the effectiveness of EPPO in mitigating reward hacking and improving RLHF performance.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 52 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

43 of 52 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  2. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  3. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
  4. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  5. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  6. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  7. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  8. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  9. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  10. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learning18 Oct
  11. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  12. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
6 more
  1. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  2. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Mistral 7Bcited by 3,065
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
13 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  2. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  3. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  4. 2019On Variational Bounds of Mutual Informationcited by 550
  5. 2018Understanding the impact of entropy on policy optimizationcited by 171
  6. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  7. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  8. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 64
  9. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 29
  10. 2024Language Models Represent Beliefs of Self and Otherscited by 29
  11. 2023BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformercited by 23
  12. 2019BERT has a Moral Compass: Improvements of ethical and moral values of machinescited by 21
  13. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  4. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  5. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
  6. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  7. 2025Think Clearly: Improving Reasoning via Redundant Token Pruningcited by 17
  8. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  9. 2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenderscited by 5
  10. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
  11. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  12. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
6 more
  1. 2025Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralizationcited by 3
  2. 2026Factored Causal Representation Learning for Robust Reward Modeling in RLHFcited by 1
  3. 2026RELISH: LLM REgression with a Latent Iterative State Headcited by 1
  4. 2025The Anti-Ouroboros Effect: Emergent Resilience in Large Language Models from Recursive Selective Feedbackcited by 0
  5. 2026DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMscited by 0
  6. 2026A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clippingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  2. 2025Learning a Pessimistic Reward Model in RLHF
  3. 2026Multimodal Reward Hacking in Reinforcement Learning
  4. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  5. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
  6. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  7. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
  8. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward