# Distributed Mixture-of-Agents for Edge Inference with Large Language Models

arXiv: https://arxiv.org/abs/2412.21200 · HTML: https://mlsanity.com/paper/2412.21200/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges](https://mlsanity.com/paper/2207.00427/) (2022)
- [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021)
- [SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents](https://mlsanity.com/paper/2411.03284/) (2024)
- [The Age of Gossip in Networks](https://mlsanity.com/paper/2102.02893/) (2021)
- [Large Language Models (LLMs) for Semantic Communication in Edge-based IoT Networks](https://mlsanity.com/paper/2407.20970/) (2024)
- [Age of Information in Gossip Networks: A Friendly Introduction and Literature Survey](https://mlsanity.com/paper/2312.16163/) (2023)

## Built on by (most cited first, 4 of 4)

- [MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs](https://mlsanity.com/paper/2507.02851/) (2025)
- [Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis](https://mlsanity.com/paper/2601.16596/) (2026)
- [Balancing Information Accuracy and Response Timeliness in Networked LLMs](https://mlsanity.com/paper/2508.02209/) (2025)
- [Beyond Scaling: Agents Are Heading to the Edge](https://mlsanity.com/paper/2605.18535/) (2026)

## Abstract (arXiv)

Mixture-of-Agents (MoA) has recently been proposed as a method to enhance performance of large language models (LLMs), enabling multiple individual LLMs to work together for collaborative inference. This collaborative approach results in improved responses to user prompts compared to relying on a single LLM. In this paper, we consider such an MoA architecture in a distributed setting, where LLMs operate on individual edge devices, each uniquely associated with a user and equipped with its own distributed computing power. These devices exchange information using decentralized gossip algorithms, allowing different device nodes to talk without the supervision of a centralized server. In the considered setup, different users have their own LLM models to address user prompts. Additionally, the devices gossip either their own user-specific prompts or augmented prompts to generate more refined answers to certain queries. User prompts are temporarily stored in the device queues when their corresponding LLMs are busy. Given the memory limitations of edge devices, it is crucial to ensure that the average queue sizes in the system remain bounded. In this paper, we address this by theoretically calculating the queuing stability conditions for the device queues under reasonable assumptions, which we validate experimentally as well. Further, we demonstrate through experiments, leveraging open-source LLMs for the implementation of distributed MoA, that certain MoA configurations produce higher-quality responses compared to others, as evaluated on AlpacaEval 2.0 benchmark. The implementation is available at: https://github.com/purbeshmitra/distributed_moa.