# Mixture of Complementary Agents for Robust LLM Ensemble

arXiv: https://arxiv.org/abs/2605.24048 · HTML: https://mlsanity.com/paper/2605.24048/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration](https://mlsanity.com/paper/2310.02170/) (2023)
- [Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models](https://mlsanity.com/paper/2311.08692/) (2023)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://mlsanity.com/paper/2305.13160/) (2023)
- [Attention Sorting Combats Recency Bias In Long Context Language Models](https://mlsanity.com/paper/2310.01427/) (2023)
- [AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2505.16400/) (2025)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?](https://mlsanity.com/paper/2502.00674/) (2025)
- [Fusing Models with Complementary Expertise](https://mlsanity.com/paper/2310.01542/) (2023)
- [LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity](https://mlsanity.com/paper/2410.03953/) (2024)
- [Relief-Based Feature Selection: Introduction and Review](https://mlsanity.com/paper/1711.08421/) (2017)
- [Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks](https://mlsanity.com/paper/2412.15238/) (2024)
- [Eliciting Informative Text Evaluations with Large Language Models](https://mlsanity.com/paper/2405.15077/) (2024)
- [Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection](https://mlsanity.com/paper/2602.08003/) (2026)
- [Enhancing LLM Code Generation with Ensembles: A Similarity-Based Selection Approach](https://mlsanity.com/paper/2503.15838/) (2025)

## Built on by (most cited first, 3 of 3)

- [Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection](https://mlsanity.com/paper/2609.38274/) (2026)
- [CopRoute: Dependence-Aware Conformal Routing for Language Models](https://mlsanity.com/paper/2610.13276/) (2026)
- [Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning](https://mlsanity.com/paper/2610.17456/) (2026)

## Abstract (arXiv)

Multi-AI collaboration, such as ensembling or debating large language models (LLMs), is a promising paradigm for aggregating information and boosting performance. A foundational step in these pipelines is to feed the responses of several proposer LLMs into a summarizer LLM, which synthesizes a better answer. However, choosing which proposers to include is non-trivial. Existing approaches primarily focus either on accuracy (picking the strongest models) or diversity (ensuring variety), and often overlook the interactions among proposers and with the summarizer. We reframe proposer selection as a combinatorial selection problem akin to feature selection, where the value of an LLM lies in its complementarity with others. However, directly applying standard feature-selection algorithms is impractical in the LLM setting due to prohibitive time complexity. Motivated by this limitation, we explore an extensive range of computationally feasible, greedy-style selection algorithms that assess complementarity using a small labeled set. Our experiments validate complementarity as a guiding principle for proposer selection and identify methods that achieve the best performance-cost trade-offs in practice.