mlsanityData to 7 Oct 2026

Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction

arXiv 2405.13432 ↗First posted 22 May 2024cs.CL cs.AIcited by 18

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Supervised fine-tuning (SFT) on instruction-following corpus is a crucial approach toward the alignment of large language models (LLMs). However, the performance of LLMs on standard knowledge and reasoning benchmarks tends to suffer from deterioration at the latter stage of the SFT process, echoing the phenomenon of alignment tax. Through our pilot study, we put a hypothesis that the data biases are probably one cause behind the phenomenon. To address the issue, we introduce a simple disperse-then-merge framework. To be concrete, we disperse the instruction-following data into portions and train multiple sub-models using different data portions. Then we merge multiple models into a single one via model merging techniques. Despite its simplicity, our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 77 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

53 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Align on the Fly: Adapting Chatbot Behavior to Established Norms26 Dec
  2. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  3. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  4. 2023LIMA: Less Is More for Alignment18 May
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
35 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2018Experience Replay for Continual Learningcited by 1,074
  13. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  14. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  15. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  16. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  17. 2023StarCoder: may the source be with you!cited by 790
  18. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  19. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  20. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  21. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  22. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  23. 2023Llemma: An Open Language Model For Mathematicscited by 385
  24. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  25. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  26. 2019Model Fusion via Optimal Transportcited by 217
  27. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  28. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  29. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  30. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
  31. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  32. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  33. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  34. 2023Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operationcited by 33
  35. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  3. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  4. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  5. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  6. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  7. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  8. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  9. 2024On the Loss of Context-awareness in General Instruction Fine-tuningcited by 10
  10. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  11. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  12. 2024Focus On This, Not That! Steering LLMs with Adaptive Feature Specificationcited by 5
6 more
  1. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  2. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 3
  3. 2024VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedbackcited by 2
  4. 2024FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimizationcited by 2
  5. 2025Conditioning LLMs to Generate Code-Switched Textcited by 2
  6. 2026Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  2. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  3. 2026DRIFT: Data Selection for LLM Instruction Tuning via On-Policy Attribution
  4. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  5. 2025Aligning Instruction Tuning with Pre-training
  6. 2024NILE: Internal Consistency Alignment in Large Language Models
  7. 2024Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data
  8. 2025LM-mixup: Text Data Augmentation via Language Model based Mixup