mlsanityData to 7 Oct 2026

WARM: On the Benefits of Weight Averaged Reward Models

arXiv 2401.12187 ↗First posted 22 Jan 2024cs.LG cs.AI cs.CLcited by 145

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences through reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit failures in the reward model (RM) to achieve seemingly high rewards without meeting the underlying objectives. We identify two primary challenges when designing RMs to mitigate reward hacking: distribution shifts during the RL process and inconsistencies in human preferences. As a solution, we propose Weight Averaged Reward Models (WARM), first fine-tuning multiple RMs, then averaging them in the weight space. This strategy follows the observation that fine-tuned weights remain linearly mode connected when sharing the same pre-training. By averaging weights, WARM improves efficiency compared to the traditional ensembling of predictions, while improving reliability under distribution shifts and robustness to preference inconsistencies. Our experiments on summarization tasks, using best-of-N and RL methods, shows that WARM improves the overall quality and alignment of LLM predictions; for example, a policy RL fine-tuned with WARM has a 79.4% win rate against a policy RL fine-tuned with a single RM.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 126 of its 152 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

126 of 152 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  2. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  3. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  5. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  6. 2023Learning Optimal Advantage from Preferences and Mistaking it for Reward3 Oct
  7. 2023Tool-Augmented Reward Modeling2 Oct
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
8 more
  1. 2023Rewarding Chatbots for Real-World Engagement with Millions of Users10 Mar
  2. 2023Active Reward Learning from Multiple Teachers2 Mar
  3. 2023On The Fragility of Learned Reward Functions9 Jan
  4. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2019Learning Human Objectives by Evaluating Hypothetical Behavior5 Dec
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 106

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2017On Calibration of Modern Neural Networkscited by 4,860
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  13. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  14. 2023PaLM 2 Technical Reportcited by 1,154
  15. 2022Editing Models with Task Arithmeticcited by 1,076
  16. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  17. 2020In Search of Lost Domain Generalizationcited by 1,042
  18. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  19. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  20. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  21. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  22. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  23. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  24. 2021Robust fine-tuning of zero-shot modelscited by 785
  25. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  26. 2023Whose Opinions Do Language Models Reflect?cited by 759
  27. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  28. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  29. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  30. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  31. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  32. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  33. 2013Domain Generalization via Invariant Feature Representationcited by 622
  34. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  35. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  36. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  37. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  38. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 69; 2025: 50; 2026: 26.

145 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  4. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  5. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  7. 2024Model merging with SVD to tie the Knotscited by 118
  8. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  9. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  10. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  11. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  12. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
38 more
  1. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  2. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  3. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  4. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  5. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  6. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  7. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  8. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  9. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  10. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  11. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  12. 2025Self-rewarding correction for mathematical reasoningcited by 67
  13. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  14. 2024What Matters for Model Merging at Scale?cited by 52
  15. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  16. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 43
  17. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  18. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  19. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  20. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  21. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  22. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  23. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  24. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  25. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  26. 2025Activation-Informed Merging of Large Language Modelscited by 29
  27. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  28. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  29. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  30. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  31. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
  32. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  33. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  34. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  35. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  36. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
  37. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  38. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  2. 2024How to Evaluate Reward Models for RLHF
  3. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  4. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  5. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  7. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  8. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models