Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation
Not on the map. Why?
Abstract (arXiv)
Model merging aims to integrate multiple task-specific models into a unified model that inherits the capabilities of the task-specific models, without additional training. Existing model merging methods often lack consideration of the varying contribution ratios of different task-specific models to the final merged model. In this paper, we propose Mixup Model Merge (M3), a simple yet effective method inspired by the randomized linear interpolation strategy from the Mixup data augmentation technique. M3 performs randomized linear interpolation in parameter space between two task-specific LLMs, where interpolation coefficients are sampled from a Beta distribution to explore diverse contribution ratios. This controllable randomness allows M3 to outperform standard equal-ratio merging by discovering better contribution ratio combinations. Extensive experiments show that M3 significantly (1) improves merged LLM performance across tasks, (2) enhances out-of-distribution and adversarial robustness, (3) outperforms the positive effects of the sparsification method DARE on model merging and can be further combined with DARE to achieve superior results, and (4) balances exploration efficiency and diversity in contribution ratios by tuning the Beta distribution's shape parameters. The code is provided in the supplementary materials.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 52 references that are in the map. First posted on arXiv 21 Feb 2025.
Builds on
33 of 52 references.
References · 33
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
21 more
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
- 2018Neural Network Acceptability Judgmentscited by 952
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
- 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
- 2021Merging Models with Fisher-Weighted Averagingcited by 613
- 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
- 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
- 2020What is being transferred in transfer learning?cited by 365
- 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
- 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
- 2023Using an LLM to Help With Code Understandingcited by 222
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
- 2018Stress Test Evaluation for Natural Language Inferencecited by 211
- 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
- 2024Evolutionary Optimization of Model Merging Recipescited by 204
- 2023Mitigating the Alignment Tax of RLHFcited by 181
- 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 77
- 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
- 2024PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuningcited by 11
Built on by
Citing papers per year: 2025: 2.
2 papers.