mlsanityData to 14 Oct 2026

MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUs

arXiv 2502.10436 ↗First posted 9 Feb 2025cs.NE cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Evolutionary model merging enables the creation of high-performing multi-task models but remains computationally prohibitive for consumer hardware. We introduce MERGE3, an efficient framework that makes evolutionary merging feasible on a single GPU by reducing fitness computation costs 50× while preserving performance. MERGE3 achieves this by Extracting a reduced dataset for evaluation, Estimating model abilities using Item Response Theory (IRT), and Evolving optimal merges via IRT-based performance estimators. Our method enables state-of-the-art multilingual and cross-lingual merging, transferring knowledge across languages with significantly lower computational overhead. We provide theoretical guarantees and an open-source library, democratizing high-quality model merging.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

31 of 49 references.

References · 31

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2023Mistral 7Bcited by 3,164
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2022Editing Models with Task Arithmeticcited by 1,120
  8. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  9. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 621
19 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  2. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  4. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  5. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  6. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  7. 2020pymoo: Multi-objective Optimization in Pythoncited by 200
  8. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  9. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  10. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  11. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  12. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  13. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  14. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  15. 2022Re-basin via implicit Sinkhorn differentiationcited by 62
  16. 2021Comparing Test Sets with Item Response Theorycited by 57
  17. 2024C2M3: Cycle-Consistent Multi-Model Mergingcited by 20
  18. 2022py-irt: A Scalable Item Response Theory Library for Pythoncited by 19
  19. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 5
  2. 2025Mergenetic: a Simple Evolutionary Model Merging Librarycited by 5
  3. 2026Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilitiescited by 5
  4. 2025Surrogate Benchmarks for Model Merging Optimizationcited by 2
  5. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
  6. 2026Zero-Shot Quantization via Weight-Space Arithmeticcited by 2
  7. 2025StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillationcited by 1
  8. 2026Model Merging: Foundations and Algorithmscited by 0
  9. 2026Bridging Spherical Black-Box Optimizerscited by 0
  10. 2026SLIM: Simplex-Lattice Interpolation Mergingcited by 0