mlsanityData to 7 Oct 2026

A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning

arXiv 2408.07057 ↗First posted 13 Aug 2024cs.LG cs.AI cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task. Model MoErging methods aim to recycle expert models to create an aggregate system with improved performance or generalization. A key component of MoErging methods is the creation of a router that decides which expert model(s) to use for a particular input or application. The promise, effectiveness, and large design space of MoErging has spurred the development of many new methods over the past few years. This rapid pace of development has made it challenging to compare different MoErging methods, which are rarely compared to one another and are often validated in different experimental setups. To remedy such gaps, we present a comprehensive survey of MoErging methods that includes a novel taxonomy for cataloging key design choices and clarifying suitable applications for each method. Apart from surveying MoErging research, we inventory software tools and applications that make use of MoErging. We additionally discuss related fields of study such as model merging, multitask learning, and mixture-of-experts models. Taken as a whole, our survey provides a unified overview of existing MoErging methods and creates a solid foundation for future work in this burgeoning field.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 106 of its 123 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

106 of 123 references.

References · 106

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2023Mistral 7Bcited by 3,065
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2022Editing Models with Task Arithmeticcited by 1,076
  13. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  14. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  15. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  16. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  17. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  18. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  19. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  20. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  21. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  22. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  23. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  24. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  25. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  26. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  27. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  28. 2024The Faiss librarycited by 370
  29. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  30. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  31. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  32. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  33. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  34. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  35. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  36. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  37. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  38. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 25; 2026: 18.

52 papers.

Sort:

By area:

  1. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  2. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 58
  3. 2024What Matters for Model Merging at Scale?cited by 52
  4. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  5. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  6. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  7. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  8. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  9. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  10. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  11. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  12. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
38 more
  1. 2025Subspace-Boosted Model Mergingcited by 9
  2. 2025Don't Throw Away Your Pretrained Modelcited by 9
  3. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 8
  4. 2025AdaRank: Adaptive Rank Pruning for Enhanced Model Mergingcited by 8
  5. 2025boldsymbolλ-Orthogonality Regularization for Compatible Representation Learningcited by 8
  6. 2024Collective Model Intelligence Requires Compatible Specializationcited by 7
  7. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  8. 2025SpectR: Dynamically Composing LM Experts with Spectral Routingcited by 6
  9. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 5
  10. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  11. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 4
  12. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  13. 2025AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learningcited by 4
  14. 2025RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Mergingcited by 3
  15. 2025Mitigating Negative Flips via Margin Preserving Trainingcited by 3
  16. 2025From Memorization to Parameter Interference: How Overtraining Experts Harms Model Mergingcited by 2
  17. 2025How does the optimizer implicitly bias the model merging loss landscape?cited by 2
  18. 2026The Appeal and Reality of Recycling LoRAs with Adaptive Mergingcited by 2
  19. 2026Modularity is the Bedrock of Natural and Artificial Intelligencecited by 2
  20. 2026Scaling Participation in Modular AI Systemscited by 2
  21. 2025Weight Weaving: Parameter Pooling for Data-Free Model Mergingcited by 1
  22. 2026The Single-Multi Evolution Loop for Self-Improving Model Collaboration Systemscited by 1
  23. 2026Unlocking the Potential of Continual Model Merging: An ODE Perspectivecited by 1
  24. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  25. 2026From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Mergingcited by 1
  26. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  27. 2024Deep Model Merging: The Sister of Neural Network Interpretability -- A Surveycited by 0
  28. 2025Out-of-Distribution Graph Models Mergingcited by 0
  29. 2025Position: Pause Recycling LoRAs and Prioritize Mechanisms to Uncover Limits and Effectivenesscited by 0
  30. 2025Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Expertscited by 0
  31. 2025Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transportcited by 0
  32. 2026GimmBO: Interactive Generative Image Model Merging via Bayesian Optimizationcited by 0
  33. 2026Fine-Grained Model Merging via Modular Expert Recombinationcited by 0
  34. 2026The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imagingcited by 0
  35. 2026Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Expertscited by 0
  36. 2026KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Trainingcited by 0
  37. 2026Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolutioncited by 0
  38. 2026StereoFactory: A Unified Merging Framework for Robust Stereo Matchingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
  2. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs
  3. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models
  4. 2025Advancing Expert Specialization for Better MoE
  5. 2024A Survey on Mixture of Experts in Large Language Models
  6. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  7. 2025MoE Pathfinder: Trajectory-driven Expert Pruning
  8. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications