mlsanityData to 14 Oct 2026

Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

arXiv 2502.00674 ↗First posted 2 Feb 2025cs.CL cs.LGcited by 54

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Ensembling outputs from diverse sources is a straightforward yet effective approach to boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggregates outputs from multiple different Large Language Models (LLMs). This paper raises the question in the context of language models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an ensemble method that aggregates outputs from only the single top-performing LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA achieves 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark, and an average of 3.8% improvement across various benchmarks, including MMLU, CRUX, and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard. To understand the effectiveness of Self-MoA, we systematically investigate the trade-off between diversity and quality of outputs under various MoA settings. We confirm that the MoA performance is rather sensitive to the quality, and mixing different LLMs often lowers the average quality of the models. To complement the study, we identify the scenarios where mixing different LLMs could be helpful. This paper further introduces a sequential version of Self-MoA, that is capable of aggregating a large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 38 of its 50 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

38 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models1 Aug
  3. 2023Universal Self-Consistency for Large Language Model Generation29 Nov

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2024Mixtral of Expertscited by 1,823
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
23 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  8. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  9. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  10. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  11. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  12. 2024Are We Done with MMLU?cited by 194
  13. 2023Mitigating the Alignment Tax of RLHFcited by 183
  14. 2024More Agents Is All You Needcited by 180
  15. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  16. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  17. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  18. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  19. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
  20. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54
  21. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  22. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 36
  23. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 31.

54 papers.

Sort:

By area:

  1. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  2. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoningcited by 27
  3. 2025Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skillscited by 24
  4. 2026Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaborationcited by 16
  5. 2025TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixturecited by 15
  6. 2025VeriMoA: A Mixture-of-Agents Framework for Spec-to-HDL Generationcited by 15
  7. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 15
  8. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 13
  9. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Saycited by 12
  10. 2025iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inferencecited by 10
  11. 2026Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensemblescited by 9
  12. 2025PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent Systemcited by 7
38 more
  1. 2025MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregationcited by 6
  2. 2025Making, not Taking, the Best of Ncited by 6
  3. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMscited by 5
  4. 2025SANet: A Semantic-aware Agentic AI Networking Framework for Cross-layer Optimization in 6Gcited by 5
  5. 2025MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMscited by 4
  6. 2026Contextual Drag: How Errors in the Context Affect LLM Reasoningcited by 4
  7. 2025Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inferencecited by 3
  8. 2025When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensemblingcited by 3
  9. 2026Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesiscited by 3
  10. 2026Mixture of Complementary Agents for Robust LLM Ensemblecited by 3
  11. 2025Multi-Agent Collaborative Intelligence: Dual-Dial Control for Reliable LLM Reasoningcited by 2
  12. 2026HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systemscited by 2
  13. 2026ReM-MoA: Reasoning Memory Sustains Mixture-of-Agents Scalingcited by 2
  14. 2026Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scalecited by 1
  15. 2026Council Mode: A Heterogeneous Multi-Agent Consensus Framework for Reducing LLM Hallucination and Biascited by 1
  16. 2026GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectivescited by 1
  17. 2026Conflict-Resilient Multi-Agent Reasoning via Signed Graph Modelingcited by 1
  18. 2026SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystemscited by 1
  19. 2026Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRAcited by 1
  20. 2025CMaP-SAM: Contraction Mapping Prior for SAM-driven Few-shot Segmentationcited by 0
  21. 2025MAC: Masked Agent Collaboration Boosts Large Language Model Medical Decision-Makingcited by 0
  22. 2025Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiencycited by 0
  23. 2025Expertise-aware Multi-LLM Recruitment and Collaboration for Medical Decision-Makingcited by 0
  24. 2025Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extractioncited by 0
  25. 2025CoLM: Collaborative Large Models via A Client-Server Paradigmcited by 0
  26. 2025WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debatecited by 0
  27. 2026Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inferencecited by 0
  28. 2026Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discoverycited by 0
  29. 2026A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliabilitycited by 0
  30. 2026Quantum-Inspired Trace-Augmented Evidence Selection for Reasoning over Structured Hypothesis Spacescited by 0
  31. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  32. 2026Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoningcited by 0
  33. 2026SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verificationcited by 0
  34. 2026What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Qualitycited by 0
  35. 2026OrchSLM: Probing the Dynamics of Small Language Model Orchestrationcited by 0
  36. 2026COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inferencecited by 0
  37. 2026Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?cited by 0
  38. 2026Towards Better Exploration in Sequential Test-Time Scalingcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Model Alignment Through Collective Intelligence of Open-Source LLMS
  2. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
  3. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild
  4. 2026MMoA: An AI-Agent framework with recurrence for Memoried Mixure-of-Agent
  5. 2026Rethinking LLM Ensembling from the Perspective of Mixture Models
  6. 2025MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
  7. 2024Distributed Mixture-of-Agents for Edge Inference with Large Language Models
  8. 2025This Is Your Doge, If It Please You: Exploring Deception and Robustness in Mixture of LLMs