mlsanityData to 7 Oct 2026

WARP: On the Benefits of Weight Averaged Rewarded Policies

arXiv 2406.16768 ↗First posted 24 Jun 2024cs.LG cs.AIcited by 42

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) aligns large language models (LLMs) by encouraging their generations to have high rewards, using a reward model trained on human preferences. To prevent the forgetting of pre-trained knowledge, RLHF usually incorporates a KL regularization; this forces the policy to remain close to its supervised fine-tuned initialization, though it hinders the reward optimization. To tackle the trade-off between KL and reward, in this paper we introduce a novel alignment strategy named Weight Averaged Rewarded Policies (WARP). WARP merges policies in the weight space at three distinct stages. First, it uses the exponential moving average of the policy as a dynamic anchor in the KL regularization. Second, it applies spherical interpolation to merge independently fine-tuned policies into a new enhanced one. Third, it linearly interpolates between this merged model and the initialization, to recover features from pre-training. This procedure is then applied iteratively, with each iteration's final model used as an advanced initialization for the next, progressively refining the KL-reward Pareto front, achieving superior rewards at fixed KL. Experiments with GEMMA policies validate that WARP improves their quality and alignment, outperforming other open-source LLMs.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 117 of its 141 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

117 of 141 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2023Nash Learning from Human Feedback1 Dec
  3. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  5. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 104

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
38 more
  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  13. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  14. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  15. 2020Learning to summarize from human feedbackcited by 2,562
  16. 2015Trust Region Policy Optimizationcited by 2,503
  17. 2016Learning without Forgettingcited by 2,465
  18. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  19. 2024Mixtral of Expertscited by 1,785
  20. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  21. 2016Concrete Problems in AI Safetycited by 1,582
  22. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  23. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  24. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  25. 2022Editing Models with Task Arithmeticcited by 1,076
  26. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  27. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  28. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  29. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  30. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  31. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  32. 2021Robust fine-tuning of zero-shot modelscited by 785
  33. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  34. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  35. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  36. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  37. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  38. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 19; 2026: 14.

42 papers.

Sort:

By area:

  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  5. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  6. 2024What Matters for Model Merging at Scale?cited by 52
  7. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  8. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  9. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 46
  10. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  11. 2025Activation-Informed Merging of Large Language Modelscited by 29
  12. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
30 more
  1. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  2. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  3. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  4. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communitiescited by 13
  5. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  6. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithmcited by 11
  7. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  8. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspectivecited by 9
  9. 2025Subspace-Boosted Model Mergingcited by 9
  10. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  11. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  12. 2025Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptationcited by 5
  13. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  14. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  15. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  16. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  17. 2026Decoupled DiLoCo for Resilient Distributed Pre-trainingcited by 4
  18. 2024Parameter-Efficient Interventions for Enhanced Model Mergingcited by 3
  19. 2026SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillationcited by 2
  20. 2025MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamicscited by 1
  21. 2025LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soupscited by 1
  22. 2026Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectorscited by 1
  23. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  24. 2026ResMerge: Residual-based Spectral Merging of Large Language Modelscited by 1
  25. 2026EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Gamescited by 1
  26. 2026Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimizationcited by 0
  27. 2026Mi:dm K 2.5 Procited by 0
  28. 2026Robust Reward Modeling for Large Language Models via Causal Decompositioncited by 0
  29. 2026REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generationcited by 0
  30. 2026Spherical Interpolation for Backward-Compatible Multimodal Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  2. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  5. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
  6. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
  7. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  8. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment