mlsanityData to 7 Oct 2026

Mixture-of-Agents Enhances Large Language Model Capabilities

arXiv 2406.04692 ↗First posted 7 Jun 2024cs.CLcited by 420

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves state-of-art performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.

Key facts. Cited by 420 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 37 of its 43 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

37 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?28 Feb
  2. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs22 Sep
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
22 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2024Mixtral of Expertscited by 1,785
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  10. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  11. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  12. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  13. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  14. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  15. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  16. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  17. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  18. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  19. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  20. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  21. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  22. 2023Fusing Models with Complementary Expertisecited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 163; 2026: 213.

420 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  4. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  5. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  6. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  7. 2025MemInsight: Autonomous Memory Augmentation for LLM Agentscited by 84
  8. 2025Latent Collaboration in Multi-Agent Systemscited by 67
  9. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  10. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 63
  11. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  12. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
38 more
  1. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  2. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  3. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  4. 2025SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoningcited by 41
  5. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  6. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  7. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  8. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoningcited by 38
  9. 2025SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoningcited by 38
  10. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  11. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  12. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 36
  13. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 35
  14. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  15. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  16. 2025Optimizing Model Selection for Compound AI Systemscited by 34
  17. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 33
  18. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  19. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  20. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  21. 2024Multi-Agent Collaboration via Cross-Team Orchestrationcited by 30
  22. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agentscited by 30
  23. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 30
  24. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  25. 2024Meta Reasoning for Large Language Modelscited by 29
  26. 2025OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi-Agent LLM Collaborationcited by 29
  27. 2025AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answeringcited by 29
  28. 2024How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?cited by 27
  29. 2025LLMs Can Generate a Better Answer by Aggregating Their Own Responsescited by 27
  30. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  31. 2024Provable Scaling Laws for the Test-Time Compute of Large Language Modelscited by 25
  32. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Informationcited by 25
  33. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMscited by 25
  34. 2026Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversitycited by 25
  35. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  36. 2025Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skillscited by 24
  37. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 23
  38. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22

50 most cited of 420 All 420 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MMoA: An AI-Agent framework with recurrence for Memoried Mixure-of-Agent
  2. 2025Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning
  3. 2026Can Small Agents Collaborate to Beat a Single Large Language Model?
  4. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results
  5. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models
  6. 2025MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
  7. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild
  8. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Tasks