# MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUs

arXiv: https://arxiv.org/abs/2502.10436 · HTML: https://mlsanity.com/paper/2502.10436/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 30 of 31)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [Evolutionary Optimization of Model Merging Recipes](https://mlsanity.com/paper/2403.13187/) (2024)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Representation Surgery for Multi-Task Model Merging](https://mlsanity.com/paper/2402.02705/) (2024)
- [Re-basin via implicit Sinkhorn differentiation](https://mlsanity.com/paper/2212.12042/) (2022)
- [Comparing Test Sets with Item Response Theory](https://mlsanity.com/paper/2106.00840/) (2021)
- [C2M3: Cycle-Consistent Multi-Model Merging](https://mlsanity.com/paper/2405.17897/) (2024)
- [py-irt: A Scalable Item Response Theory Library for Python](https://mlsanity.com/paper/2203.01282/) (2022)

## Built on by (most cited first, 10 of 10)

- [MASS: MoErging through Adaptive Subspace Selection](https://mlsanity.com/paper/2504.05342/) (2025)
- [Mergenetic: a Simple Evolutionary Model Merging Library](https://mlsanity.com/paper/2505.11427/) (2025)
- [Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities](https://mlsanity.com/paper/2603.10396/) (2026)
- [Surrogate Benchmarks for Model Merging Optimization](https://mlsanity.com/paper/2509.02555/) (2025)
- [Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Models](https://mlsanity.com/paper/2511.05171/) (2025)
- [Zero-Shot Quantization via Weight-Space Arithmetic](https://mlsanity.com/paper/2604.03420/) (2026)
- [StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation](https://mlsanity.com/paper/2506.04567/) (2025)
- [Model Merging: Foundations and Algorithms](https://mlsanity.com/paper/2605.01580/) (2026)
- [Bridging Spherical Black-Box Optimizers](https://mlsanity.com/paper/2606.25761/) (2026)
- [SLIM: Simplex-Lattice Interpolation Merging](https://mlsanity.com/paper/2610.01037/) (2026)

## Abstract (arXiv)

Evolutionary model merging enables the creation of high-performing multi-task models but remains computationally prohibitive for consumer hardware. We introduce MERGE3, an efficient framework that makes evolutionary merging feasible on a single GPU by reducing fitness computation costs 50× while preserving performance. MERGE3 achieves this by Extracting a reduced dataset for evaluation, Estimating model abilities using Item Response Theory (IRT), and Evolving optimal merges via IRT-based performance estimators. Our method enables state-of-the-art multilingual and cross-lingual merging, transferring knowledge across languages with significantly lower computational overhead. We provide theoretical guarantees and an open-source library, democratizing high-quality model merging.