mlsanityData to 7 Oct 2026

Arcee's MergeKit: A Toolkit for Merging Large Language Models

arXiv 2403.13257 ↗First posted 20 Mar 2024cs.CL cs.AI cs.LGcited by 189

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid expansion of the open-source language model landscape presents an opportunity to merge the competencies of these model checkpoints by combining their parameters. Advances in transfer learning, the process of fine-tuning pretrained models for specific tasks, has resulted in the development of vast amounts of task-specific models, typically specialized in individual tasks and unable to utilize each other's strengths. Model merging facilitates the creation of multitask models without the need for additional training, offering a promising avenue for enhancing model performance and versatility. By preserving the intrinsic capabilities of the original models, model merging addresses complex challenges in AI - including the difficulties of catastrophic forgetting and multitask learning. To support this expanding area of research, we introduce MergeKit, a comprehensive, open-source library designed to facilitate the application of model merging strategies. MergeKit offers an extensible framework to efficiently merge models on any hardware, providing utility to researchers and practitioners. To date, thousands of models have been merged by the open-source community, leading to the creation of some of the worlds most powerful open-source model checkpoints, as assessed by the Open LLM Leaderboard. The library is accessible at https://github.com/arcee-ai/MergeKit.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Builds on 40 of its 58 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

40 of 58 references.

References · 40

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
28 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  6. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  7. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  8. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  10. 2020What is being transferred in transfer learning?cited by 365
  11. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  13. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  14. 2019Model Fusion via Optimal Transportcited by 217
  15. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  16. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  17. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  18. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  19. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  20. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  21. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  22. 2022Fusing finetuned models for better pretrainingcited by 110
  23. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  24. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  25. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  26. 2024LLM Augmented LLMs: Expanding Capabilities through Compositioncited by 35
  27. 2024Merging Text Transformer Models from Different Initializationscited by 16
  28. 2024OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 50; 2025: 81; 2026: 58.

189 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  3. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  4. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  5. 2024WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Modelscited by 90
  6. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 67
  7. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  8. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  9. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  10. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  11. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
  12. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
38 more
  1. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  2. 2025Activation-Informed Merging of Large Language Modelscited by 29
  3. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  4. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimizationcited by 25
  5. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  6. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  7. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  8. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  9. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  10. 2024LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generationcited by 19
  11. 2024PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarkscited by 18
  12. 2024Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Mergingcited by 17
  13. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  14. 2024SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Mergingcited by 16
  15. 2025Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLMcited by 16
  16. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  17. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 14
  18. 2025Nature-Inspired Population-Based Evolution of Large Language Modelscited by 14
  19. 2025SoK: Large Language Model Copyright Auditing via Fingerprintingcited by 14
  20. 2025Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performancecited by 13
  21. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  22. 2025The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMscited by 12
  23. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglementcited by 11
  24. 2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11
  25. 2025Soup to go: mitigating forgetting during continual learning with model averagingcited by 11
  26. 2025Towards Watermarking of Open-Source LLMscited by 11
  27. 2025CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoorcited by 11
  28. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
  29. 2024It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimizationcited by 10
  30. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10
  31. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  32. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wildcited by 9
  33. 2025Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging -- An Open Recipecited by 9
  34. 2025MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUscited by 9
  35. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  36. 2025Subspace-Boosted Model Mergingcited by 9
  37. 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistantscited by 9
  38. 2025Don't Throw Away Your Pretrained Modelcited by 9

50 most cited of 189 All 189 (JSON)