mlsanityData to 7 Oct 2026

ODIN: Disentangled Reward Mitigates Hacking in RLHF

arXiv 2402.07319 ↗First posted 11 Feb 2024cs.LG cs.AI cs.CLcited by 133

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this work, we study the issue of reward hacking on the response length, a challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on LLMs. A well-formatted, verbose but less helpful response from the LLMs can often deceive LLMs or even human evaluators to achieve high scores. The same issue also holds for some reward models in RL. To address the challenges in both training and evaluation, we establish a more reliable evaluation protocol for comparing different training configurations, which inspects the trade-off between LLM evaluation score and response length obtained by varying training hyperparameters. Based on this evaluation, we conduct large-scale studies, where the results shed insights into the efficacy of hyperparameters and tricks used in RL on mitigating length bias. We further propose to improve the reward model by jointly training two linear heads on shared feature representations to predict the rewards, one trained to correlate with length, and the other trained to decorrelate with length and therefore focus more on the actual content. We then discard the length head in RL to prevent reward hacking on length. Experiments demonstrate that our approach almost eliminates the reward correlation with length, and improves the obtained policy by a significant margin.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 59 references that are in the map. First posted on arXiv 11 Feb 2024.

Builds on

44 of 59 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  5. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  8. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  9. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  10. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
4 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
16 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  9. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  11. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  12. 2021Unsolved Problems in ML Safetycited by 291
  13. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  14. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  15. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  16. 2016Length bias in Encoder Decoder Models and a Case for Global Conditioningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 56; 2026: 46.

133 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  4. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  5. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  6. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  7. 2025Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Provingcited by 126
  8. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  9. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  10. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  11. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  12. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
38 more
  1. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  2. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  3. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  6. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  7. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  8. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 46
  9. 2024Following Length Constraints in Instructionscited by 45
  10. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  11. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  12. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  13. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 39
  14. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  15. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  16. 2024Reward-Robust RLHF in LLMscited by 32
  17. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  18. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  19. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  20. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  21. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  22. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  23. 2024Rational Metareasoning for Large Language Modelscited by 20
  24. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  25. 2025Robust Reward Modeling via Causal Rubricscited by 18
  26. 2025Adversarial Training of Reward Modelscited by 17
  27. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  28. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  29. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
  30. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 15
  31. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  32. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  33. 2025Robust Preference Optimization via Dynamic Target Marginscited by 13
  34. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  35. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  36. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  37. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  38. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  2. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  3. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2026Multimodal Reward Hacking in Reinforcement Learning
  6. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  7. 2024How to Evaluate Reward Models for RLHF
  8. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards