mlsanityData to 14 Oct 2026

Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment

arXiv 2405.17931 ↗First posted 28 May 2024cs.CL cs.LGcited by 28

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Effectively aligning Large Language Models (LLMs) with human-centric values while preventing the degradation of abilities acquired through Pre-training and Supervised Fine-tuning (SFT) poses a central challenge in Reinforcement Learning from Human Feedback (RLHF). In this paper, we first discover that interpolating RLHF and SFT model parameters can adjust the trade-off between human preference and basic capabilities, thereby reducing the alignment tax at the cost of alignment reward. Inspired by this, we propose integrating the RL policy and SFT models at each optimization step in RLHF to continuously regulate the training direction, introducing the Online Merging Optimizer. Specifically, we merge gradients with the parameter differences between SFT and pretrained models, effectively steering the gradient towards maximizing rewards in the direction of SFT optimization. We demonstrate that our optimizer works well with different LLM families, such as Qwen and LLaMA, across various model sizes ranging from 1.8B to 8B, various RLHF algorithms like DPO and KTO, and existing model merging methods. It significantly enhances alignment reward while mitigating alignment tax, achieving higher overall performance across 14 benchmarks.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 65 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

56 of 65 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Model Extrapolation Expedites Alignment25 Apr
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Direct Preference Optimization with an Offset16 Feb
  6. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  7. 2023Language Model Alignment with Elastic Reset6 Dec
  8. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Mitigating the Alignment Tax of RLHF12 Sep
  11. 2023Preference Ranking Optimization for Human Alignment30 Jun
  12. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2016Learning without Forgettingcited by 2,496
  12. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
31 more
  1. 2017Decoupled Weight Decay Regularizationcited by 1,831
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  7. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  8. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  10. 2018CoQA: A Conversational Question Answering Challengecited by 683
  11. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  13. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  14. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  15. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  16. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  17. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  18. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  19. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  20. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  21. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  22. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  23. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  24. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  25. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  26. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  27. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  28. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  29. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  30. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  31. 2022HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 7; 2026: 3.

28 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2024Qwen2 Technical Reportcited by 2,120
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  4. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  5. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  6. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  7. 2025A Survey of Direct Preference Optimizationcited by 36
  8. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 31
  9. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  10. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  11. 2024Superficial Safety Alignment Hypothesiscited by 14
  12. 2025Mapping Post-Training Forgetting in Language Models at Scalecited by 11
16 more
  1. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  2. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  3. 2024Merging Improves Self-Critique Against Jailbreak Attackscited by 8
  4. 2025CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Mergingcited by 8
  5. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalizationcited by 6
  6. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  7. 2024Aqulia-Med LLM: Pioneering Full-Process Open-Source Medical Language Modelscited by 4
  8. 2024Continual SFT Matches Multimodal RLHF with Negative Supervisioncited by 4
  9. 2024FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimizationcited by 2
  10. 2024CareBot: A Pioneering Full-Process Open-Source Medical Language Modelcited by 2
  11. 2025MOSLIM:Align with diverse preferences in prompts through reward classificationcited by 2
  12. 2026EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMscited by 2
  13. 2024Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Modelcited by 0
  14. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimizationcited by 0
  15. 2026Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learningcited by 0
  16. 2026Toward a Theory of Value in AI Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
  2. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  3. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  4. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  5. 2025A Principled Loss Function for Direct Language Model Alignment
  6. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  7. 2024Accelerated Preference Optimization for Large Language Model Alignment
  8. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs