# Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

arXiv: https://arxiv.org/abs/2502.00674 · HTML: https://mlsanity.com/paper/2502.00674/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 38 of its 50 references that are in the map.
- First posted on arXiv 2 Feb 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [Mitigating the Alignment Tax of RLHF](https://mlsanity.com/paper/2309.06256/) (2023)
- [More Agents Is All You Need](https://mlsanity.com/paper/2402.05120/) (2024)
- [Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?](https://mlsanity.com/paper/2402.18272/) (2024)

## Built on by (most cited first, 30 of 54)

- [The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants](https://mlsanity.com/paper/2505.19797/) (2025)
- [Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning](https://mlsanity.com/paper/2504.05047/) (2025)
- [Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills](https://mlsanity.com/paper/2503.05641/) (2025)
- [Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration](https://mlsanity.com/paper/2604.17148/) (2026)
- [TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture](https://mlsanity.com/paper/2510.01279/) (2025)
- [VeriMoA: A Mixture-of-Agents Framework for Spec-to-HDL Generation](https://mlsanity.com/paper/2510.27617/) (2025)
- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models](https://mlsanity.com/paper/2606.27288/) (2026)
- [Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say](https://mlsanity.com/paper/2509.21164/) (2025)
- [iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference](https://mlsanity.com/paper/2511.11306/) (2025)
- [Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles](https://mlsanity.com/paper/2607.20768/) (2026)
- [PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent System](https://mlsanity.com/paper/2509.24855/) (2025)
- [MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation](https://mlsanity.com/paper/2506.17113/) (2025)
- [Making, not Taking, the Best of N](https://mlsanity.com/paper/2510.00931/) (2025)
- [LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs](https://mlsanity.com/paper/2507.16809/) (2025)
- [SANet: A Semantic-aware Agentic AI Networking Framework for Cross-layer Optimization in 6G](https://mlsanity.com/paper/2512.22579/) (2025)
- [MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs](https://mlsanity.com/paper/2507.02851/) (2025)
- [Contextual Drag: How Errors in the Context Affect LLM Reasoning](https://mlsanity.com/paper/2602.04288/) (2026)
- [Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inference](https://mlsanity.com/paper/2509.07571/) (2025)
- [When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling](https://mlsanity.com/paper/2510.15346/) (2025)
- [Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis](https://mlsanity.com/paper/2601.16596/) (2026)
- [Mixture of Complementary Agents for Robust LLM Ensemble](https://mlsanity.com/paper/2605.24048/) (2026)
- [Multi-Agent Collaborative Intelligence: Dual-Dial Control for Reliable LLM Reasoning](https://mlsanity.com/paper/2510.04488/) (2025)
- [HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems](https://mlsanity.com/paper/2602.20229/) (2026)
- [ReM-MoA: Reasoning Memory Sustains Mixture-of-Agents Scaling](https://mlsanity.com/paper/2606.24437/) (2026)
- [Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scale](https://mlsanity.com/paper/2601.01330/) (2026)
- [Council Mode: A Heterogeneous Multi-Agent Consensus Framework for Reducing LLM Hallucination and Bias](https://mlsanity.com/paper/2604.02923/) (2026)
- [GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives](https://mlsanity.com/paper/2605.09027/) (2026)
- [Conflict-Resilient Multi-Agent Reasoning via Signed Graph Modeling](https://mlsanity.com/paper/2605.19418/) (2026)
- [SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystems](https://mlsanity.com/paper/2606.03544/) (2026)

## Abstract (arXiv)

Ensembling outputs from diverse sources is a straightforward yet effective approach to boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggregates outputs from multiple different Large Language Models (LLMs). This paper raises the question in the context of language models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an ensemble method that aggregates outputs from only the single top-performing LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA achieves 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark, and an average of 3.8% improvement across various benchmarks, including MMLU, CRUX, and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard. To understand the effectiveness of Self-MoA, we systematically investigate the trade-off between diversity and quality of outputs under various MoA settings. We confirm that the MoA performance is rather sensitive to the quality, and mixing different LLMs often lowers the average quality of the models. To complement the study, we identify the scenarios where mixing different LLMs could be helpful. This paper further introduces a sequential version of Self-MoA, that is capable of aggregating a large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.