mlsanityData to 14 Oct 2026

Harnessing Multiple Large Language Models: A Survey on LLM Ensemble

arXiv 2502.18036 ↗First posted 25 Feb 2025cs.CLcited by 86

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LLM Ensemble -- which involves the comprehensive use of multiple large language models (LLMs), each aimed at handling user queries during downstream inference, to benefit from their individual strengths -- has gained substantial attention recently. The widespread availability of LLMs, coupled with their varying strengths and out-of-the-box usability, has profoundly advanced the field of LLM Ensemble. This paper presents the first systematic review of recent developments in LLM Ensemble. First, we introduce our taxonomy of LLM Ensemble and discuss several related research problems. Then, we provide a more in-depth classification of the methods under the broad categories of "ensemble-before-inference, ensemble-during-inference, ensemble-after-inference'', and review all relevant methods. Finally, we introduce related benchmarks and applications, summarize existing studies, and suggest several future research directions. A curated list of papers on LLM Ensemble is available at https://github.com/junchenzhi/Awesome-LLM-Ensemble.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 70 of its 117 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

70 of 117 references.

References · 70

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  7. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  11. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 372
  12. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
38 more
  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  3. 2022Relative representations enable zero-shot latent space communicationcited by 191
  4. 2024More Agents Is All You Needcited by 180
  5. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  6. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  7. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
  8. 2023AutoMix: Automatically Mixing Language Modelscited by 125
  9. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
  10. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 96
  11. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  12. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  13. 2023Deep Model Fusion: A Surveycited by 91
  14. 2024A Unified Approach to Routing and Cascading for LLMscited by 80
  15. 2024Decoding-time Realignment of Language Modelscited by 67
  16. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 67
  17. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  18. 2022A Survey on Programmatic Weak Supervisioncited by 66
  19. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 62
  20. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 60
  21. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
  22. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  23. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  24. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 49
  25. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  26. 2024Bridging the Gap between Different Vocabularies for LLM Ensemblecited by 45
  27. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  28. 2023Getting MoRE out of Mixture of Language Model Reasoning Expertscited by 44
  29. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  30. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42
  31. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 41
  32. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoningcited by 40
  33. 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 35
  34. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  35. 2025LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routingcited by 34
  36. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  37. 2024Smoothie: Label Free Language Model Routingcited by 31
  38. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 49; 2026: 37.

86 papers.

Sort:

By area:

  1. 2025Correlated Errors in Large Language Modelscited by 45
  2. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  3. 2025Parallel Scaling Law for Language Modelscited by 30
  4. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  5. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Informationcited by 26
  6. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  7. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  8. 2025Speculative Actions: A Lossless Framework for Faster Agentic Systemscited by 22
  9. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 19
  10. 2025Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluationscited by 16
  11. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 14
  12. 2025Towards Responsible and Explainable AI Agents with Consensus-Driven Reasoningcited by 14
38 more
  1. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  2. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  3. 2025Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routerscited by 11
  4. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  5. 2026Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selectioncited by 11
  6. 2025MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Modelscited by 10
  7. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  8. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  9. 2026Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensemblescited by 9
  10. 2025INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profilingcited by 8
  11. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8
  12. 2025LLMBoost: Make Large Language Models Stronger with Boostingcited by 8
  13. 2025RLAE: Reinforcement Learning-Assisted Ensemble for LLMscited by 7
  14. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  15. 2025Semantic Agreement Enables Efficient Open-Ended LLM Cascadescited by 7
  16. 2026RACER: Risk-Aware Calibrated Efficient Routing for Large Language Modelscited by 7
  17. 2025Uncertainty Quantification for Retrieval-Augmented Reasoningcited by 6
  18. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  19. 2025DiSRouter: Distributed Self-Routing for LLM Selectionscited by 5
  20. 2026Do explanations generalize across large reasoning models?cited by 5
  21. 2025Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLMcited by 4
  22. 2025LLMRank: Understanding LLM Strengths for Model Routingcited by 4
  23. 2025When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?cited by 4
  24. 2025Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Processcited by 4
  25. 2026From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialoguecited by 4
  26. 2025MArgE: Meshing Argumentative Evidence from Multiple Large Language Models for Justifiable Claim Verificationcited by 3
  27. 2025CARGO: A Framework for Confidence-Aware Routing of Large Language Modelscited by 3
  28. 2025Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensemblingcited by 3
  29. 2025LLM Chemistry Estimation for Multi-LLM Recommendationcited by 3
  30. 2025When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensemblingcited by 3
  31. 2025The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Modelscited by 3
  32. 2026Next Generation Active Learning: Mixture of LLMs in the Loopcited by 3
  33. 2026Controlling Reading Ease with Gaze-Guided Text Generationcited by 3
  34. 2026Mixture of Complementary Agents for Robust LLM Ensemblecited by 3
  35. 2025EL4NER: Ensemble Learning for Named Entity Recognition via Multiple Small-Parameter Large Language Modelscited by 2
  36. 2025WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agentcited by 2
  37. 2025Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Modelscited by 2
  38. 2026RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systemscited by 2

50 most cited of 86 All 86 (JSON)