Data to 7 Oct 2026

Mitigating the Alignment Tax of RLHF

arXiv 2309.06256 ↗First posted 12 Sep 2023cs.LGcited by 181

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

LLMs acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, which is also known as the alignment tax. To investigate alignment tax, we conducted experiments with existing RLHF algorithms using OpenLLaMA-3B, which revealed a pronounced alignment tax in NLP tasks. Whereas, despite various techniques to mitigate forgetting, they are often at odds with the RLHF performance, leading to a trade-off between alignment performance and forgetting mitigation, leading to an alignment-forgetting trade-off. In this paper we show that model averaging, which simply interpolates between pre and post RLHF model weights, surprisingly achieves the most strongest alignment-forgetting Pareto front among a wide range of competing methods. To understand its effectiveness, we offer theoretical insights into model averaging, revealing that it enhances performance Pareto front by increasing feature diversity on the layers where tasks share overlapped feature spaces. Empirical evidence corroborates our analysis by showing the benefits of averaging low-level transformer layers. Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different alignment-forgetting trade-offs, we propose Heterogeneous Model Averaging (HMA) to Heterogeneously find various combination ratios of model layers. HMA seeks to maximize the alignment performance while incurring minimal alignment tax. Moreover, we validate HMA's performance across a range of RLHF algorithms over OpenLLaMA-3B and further extend our findings to Mistral-7B which is evaluated by open-sourced preference model and GPT4. Code available here: https://github.com/avalonstrel/Mitigating-the-Alignment-Tax-of-RLHF.git.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 81 of its 117 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

81 of 117 references.

Closest ancestors · 3

Same direction, newest first

Other references · 78

2015Deep Residual Learning for Image Recognitioncited by 51,809
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Language Models are Few-Shot Learnerscited by 32,199
2023GPT-4 Technical Reportcited by 18,132
2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
2017Proximal Policy Optimization Algorithmscited by 14,441
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2016Overcoming catastrophic forgetting in neural networkscited by 4,718

38 more

2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
2017Deep reinforcement learning from human preferencescited by 3,846
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
2013Visualizing and Understanding Convolutional Networkscited by 2,278
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
2017Continual Learning with Deep Generative Replaycited by 1,152
2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
2018Experience Replay for Continual Learningcited by 1,074
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2018Efficient Lifelong Learning with A-GEMcited by 1,044
2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
2021Robust fine-tuning of zero-shot modelscited by 785
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2022Galactica: A Large Language Model for Sciencecited by 736
2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
2018Progress & Compress: A scalable framework for continual learningcited by 516
2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
2015Understanding Neural Networks Through Deep Visualizationcited by 332
2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
2021Recursively Summarizing Books with Human Feedbackcited by 287
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 44; 2025: 69; 2026: 65.

181 papers.

Sort:

By area:

2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
2024Continual Learning for Large Language Models: A Surveycited by 144
2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
2025HalluLens: LLM Hallucination Benchmarkcited by 88
2024A Survey on Self-Evolution of Large Language Modelscited by 86
2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
2023Alignment for Honestycited by 72

38 more

2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
2024Aligning LLMs with Individual Preferences via Interactioncited by 63
2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 48
2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
2024Model Extrapolation Expedites Alignmentcited by 37
2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
2025A Survey of Direct Preference Optimizationcited by 35
2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
2024Creativity Has Left the Chat: The Price of Debiasing Language Modelscited by 29
2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
2024How to Merge Your Multimodal Models Over Time?cited by 23
2025Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoningcited by 23
2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuningcited by 19
2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Basescited by 17
2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
2024Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answerscited by 15
2023CITING: Large Language Models Create Curriculum for Instruction Tuningcited by 14
2024Superficial Safety Alignment Hypothesiscited by 14
2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 14
2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
2025ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialectscited by 11
2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11

50 most cited of 181 All 181 (JSON)

Similar, not cited

Close in wording, no citation either way