mlsanityData to 13 Oct 2026

SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents

arXiv 2411.03284 ↗First posted 5 Nov 2024cs.AI cs.CL cs.MAcited by 30

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

While multi-agent systems have been shown to significantly enhance the performance of Large Language Models (LLMs) across various tasks and applications, the dense interaction between scaling agents potentially hampers their efficiency and diversity. To address these challenges, we draw inspiration from the sparse mixture-of-agents (SMoE) and propose a sparse mixture-of-agents (SMoA) framework to improve the efficiency and diversity of multi-agent LLMs. Unlike completely connected structures, SMoA introduces novel Response Selection and Early Stopping mechanisms to sparsify information flows among individual LLM agents, striking a balance between performance and efficiency. Additionally, inspired by the expert diversity principle in SMoE frameworks for workload balance between experts, we assign distinct role descriptions to each LLM agent, fostering diverse and divergent thinking. Extensive experiments on reasoning, alignment, and fairness benchmarks demonstrate that SMoA achieves performance comparable to traditional mixture-of-agents approaches but with significantly lower computational costs. Further analysis reveals that SMoA is more stable, has a greater capacity to scale, and offers considerable potential through hyper-parameter optimization. Code and data will be available at: https://github.com/David-Li0406/SMoA.

Key facts. Cited by 30 papers in the map (data through 13 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 40 of its 57 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

40 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains18 Jul
  2. 2024Scaling Large Language Model-based Multi-Agent Collaboration11 Jun
  3. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  4. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration3 Oct
  5. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2024The Llama 3 Herd of Modelscited by 14,486
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  6. 2023Qwen Technical Reportcited by 3,310
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  8. 2024Qwen2 Technical Reportcited by 2,102
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,017
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,758
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
22 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,091
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,001
  3. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 448
  4. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 255
  6. 2021Hash Layers For Large Sparse Modelscited by 237
  7. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  8. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  9. 2023Learning From Mistakes Makes LLM Better Reasonercited by 98
  10. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 92
  11. 2024Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoningcited by 69
  12. 2024"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Modelscited by 60
  13. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  14. 2021Diverse Adversaries for Mitigating Bias in Trainingcited by 48
  15. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  16. 2023A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detectioncited by 42
  17. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  18. 2021First the worst: Finding better gender translations during beam searchcited by 29
  19. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinkingcited by 19
  20. 2024Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?cited by 17
  21. 2023Multi-level Contrastive Learning for Script-based Character Understandingcited by 15
  22. 2023Ziya2: Data-centric Learning is All LLMs Needcited by 15

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 10.

30 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 483
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 48
  3. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  4. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  5. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  6. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 21
  7. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  8. 2025Learning from Peers in Reasoning Modelscited by 16
  9. 2025Literature Review Of Multi-Agent Debate For Problem-Solvingcited by 15
  10. 2025Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillationcited by 15
  11. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
  12. 2025S-DAG: A Subject-Based Directed Acyclic Graph for Multi-Agent Heterogeneous Reasoningcited by 9
18 more
  1. 2025Towards Multi-Agent Reasoning Systems for Collaborative Expertise Delegation: An Exploratory Design Studycited by 8
  2. 2025Towards Cognitive Synergy in LLM-Based Multi-Agent Systems: Integrating Theory of Mind and Critical Evaluationcited by 8
  3. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Modelscited by 7
  4. 2025Who's Your Judge? On the Detectability of LLM-Generated Judgmentscited by 7
  5. 2025MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregationcited by 6
  6. 2025Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstructioncited by 6
  7. 2026EvoMAS: Evolutionary Generation of Multi-Agent Systemscited by 5
  8. 2024Distributed Mixture-of-Agents for Edge Inference with Large Language Modelscited by 4
  9. 2025MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMscited by 4
  10. 2026Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesiscited by 3
  11. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
  12. 2025Balancing Information Accuracy and Response Timeliness in Networked LLMscited by 2
  13. 2026Dynamic Trust-Aware Sparse Communication Topology for LLM-Based Multi-Agent Consensuscited by 2
  14. 2026Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoningcited by 1
  15. 2026CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologiescited by 1
  16. 2026OptiMAS: Automatically Optimize Multi-Agent Systemcited by 0
  17. 2026OrchSLM: Probing the Dynamics of Small Language Model Orchestrationcited by 0
  18. 2026Topological Coherence for Self-evolving Multi-agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation
  2. 2026Can Small Agents Collaborate to Beat a Single Large Language Model?
  3. 2026MMoA: An AI-Agent framework with recurrence for Memoried Mixure-of-Agent
  4. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
  5. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
  6. 2024CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models
  7. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
  8. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challenges