mlsanityData to 7 Oct 2026

Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models

arXiv 2407.06089 ↗First posted 8 Jul 2024cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

The remarkable success of Large Language Models (LLMs) has ushered natural language processing (NLP) research into a new era. Despite their diverse capabilities, LLMs trained on different corpora exhibit varying strengths and weaknesses, leading to challenges in maximizing their overall efficiency and versatility. To address these challenges, recent studies have explored collaborative strategies for LLMs. This paper provides a comprehensive overview of this emerging research area, highlighting the motivation behind such collaborations. Specifically, we categorize collaborative strategies into three primary approaches: Merging, Ensemble, and Cooperation. Merging involves integrating multiple LLMs in the parameter space. Ensemble combines the outputs of various LLMs. Cooperation} leverages different LLMs to allow full play to their diverse capabilities for specific tasks. We provide in-depth introductions to these methods from different perspectives and discuss their potential applications. Additionally, we outline future research directions, hoping this work will catalyze further studies on LLM collaborations and paving the way for advanced NLP applications.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 155 of its 193 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

155 of 193 references.

References · 155

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  13. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  14. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  15. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  16. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  17. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  18. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  19. 2023BloombergGPT: A Large Language Model for Financecited by 923
  20. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  21. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  22. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  23. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  24. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  25. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  26. 2022A Survey on In-context Learningcited by 717
  27. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  28. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  29. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  30. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  31. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  32. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  33. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  34. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  35. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  36. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  37. 2023Llemma: An Open Language Model For Mathematicscited by 385
  38. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384

50 most cited of 155 All 155 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 28; 2026: 10.

43 papers.

Sort:

By area:

  1. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  2. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  3. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  4. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 36
  5. 2025Optimizing Model Selection for Compound AI Systemscited by 34
  6. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 32
  7. 2025Neighboring Autoregressive Modeling for Efficient Visual Generationcited by 29
  8. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Informationcited by 25
  9. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  10. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  11. 2025Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluationscited by 16
  12. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
31 more
  1. 2025Fast Large Language Model Collaborative Decoding via Speculationcited by 12
  2. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  3. 2025Subspace-Boosted Model Mergingcited by 9
  4. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  5. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  6. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  7. 2025LLMBoost: Make Large Language Models Stronger with Boostingcited by 8
  8. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truthcited by 7
  9. 2025Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Expertscited by 7
  10. 2024Enhancing Answer Reliability Through Inter-Model Consensus of Large Language Modelscited by 6
  11. 2025VL-RouterBench: A Benchmark for Vision-Language Model Routingcited by 6
  12. 2025Position: Enough of Scaling LLMs! Lets Focus on Downscalingcited by 5
  13. 2025RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Priorcited by 5
  14. 2025Improving Model Alignment Through Collective Intelligence of Open-Source LLMScited by 4
  15. 2025Characterizing the Investigative Methods of Fictional Detectives with Large Language Modelscited by 3
  16. 2026Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMscited by 3
  17. 2024Exploring Model Kinship for Merging Large Language Modelscited by 2
  18. 2025LLM Chemistry Estimation for Multi-LLM Recommendationcited by 2
  19. 2026Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Mergingcited by 2
  20. 2026ReasonAny: Incorporating Reasoning Capability to Any Model via Simple and Effective Model Mergingcited by 1
  21. 2026Rethinking LLM Ensembling from the Perspective of Mixture Modelscited by 1
  22. 2026Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspectivecited by 1
  23. 2024Large Language Models Merging for Enhancing the Link Stealing Attack on Graph Neural Networkscited by 0
  24. 2025A Trustworthy Multi-LLM Network: Challenges,Solutions, and A Use Casecited by 0
  25. 2025AI-powered virtual eye: perspective, challenges and opportunitiescited by 0
  26. 2025PSO-Merging: Merging Models Based on Particle Swarm Optimizationcited by 0
  27. 2025Knots: A Large-Scale Multi-Agent Enhanced Expert-Annotated Dataset and LLM Prompt Optimization for NOTAM Semantic Parsingcited by 0
  28. 2026Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Expertscited by 0
  29. 2026SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systemscited by 0
  30. 2026CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoningcited by 0
  31. 2026Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selectioncited by 0