mlsanityData to 7 Oct 2026

Evolutionary Optimization of Model Merging Recipes

arXiv 2403.13187 ↗First posted 19 Mar 2024cs.NEcited by 204

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have become increasingly capable, but their development often requires substantial computational resources. While model merging has emerged as a cost-effective promising approach for creating new models by combining existing ones, it currently relies on human intuition and domain knowledge, limiting its potential. Here, we propose an evolutionary approach that overcomes this limitation by automatically discovering effective combinations of diverse open-source models, harnessing their collective intelligence without requiring extensive additional training data or compute. Our approach operates in both parameter space and data flow space, allowing for optimization beyond just the weights of the individual models. This approach even facilitates cross-domain merging, generating models like a Japanese LLM with Math reasoning capabilities. Surprisingly, our Japanese Math LLM achieved state-of-the-art performance on a variety of established Japanese LLM benchmarks, even surpassing models with significantly more parameters, despite not being explicitly trained for such tasks. Furthermore, a culturally-aware Japanese VLM generated through our approach demonstrates its effectiveness in describing Japanese culture-specific content, outperforming previous Japanese VLMs. This work not only contributes new state-of-the-art models back to the open-source community, but also introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Mar 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 70; 2025: 75; 2026: 59.

204 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  4. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  5. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  6. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  7. 2024What Matters for Model Merging at Scale?cited by 52
  8. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  9. 2024Transformer Layers as Painterscited by 50
  10. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  11. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
  12. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
38 more
  1. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  2. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 41
  3. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  4. 2024Model Extrapolation Expedites Alignmentcited by 37
  5. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  6. 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 34
  7. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 30
  8. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  9. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 28
  10. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  11. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 27
  12. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  13. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  14. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  15. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  16. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  17. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  18. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  19. 2024JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluationcited by 19
  20. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  21. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
  22. 2024PLeaS -- Merging Models with Permutations and Least Squarescited by 16
  23. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  24. 2025Personality Vector: Modulating Personality of Large Language Models by Model Mergingcited by 16
  25. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  26. 2025Nature-Inspired Population-Based Evolution of Large Language Modelscited by 14
  27. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  28. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
  29. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanesecited by 12
  30. 2024Realistic Evaluation of Model Merging for Compositional Generalizationcited by 12
  31. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  32. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  33. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 11
  34. 2024It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimizationcited by 10
  35. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10
  36. 2024Agent Skill Acquisition for Large Language Models via CycleQDcited by 10
  37. 2025Model Merging Scaling Laws in Large Language Modelscited by 10
  38. 2024Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Designcited by 9

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
  2. 2025The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs
  3. 2024MoD: A Distribution-Based Approach for Merging Large Language Models
  4. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
  5. 2025Evolutionary thoughts: integration of large language models and evolutionary algorithms
  6. 2026A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMDelta Integration into Upcycled MoE
  7. 2025The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging
  8. 2026Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective