mlsanityData to 7 Oct 2026

Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberation

arXiv 2601.16863 ↗First posted 23 Jan 2026cs.AI cs.LG cs.MAcited by 0

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

Abstract (arXiv)

This paper introduces the N-Way Self-Evaluating Deliberation (NSED) protocol, a Runtime Mixture-of-Models (MoM) architecture that constructs emergent composite models from a plurality of distinct expert agents. Unlike traditional Mixture-of-Experts (MoE) which rely on static gating networks, NSED employs a Dynamic Expertise Broker - a runtime optimization engine that treats model selection as a variation of the Knapsack Problem, binding heterogeneous checkpoints to functional roles based on live telemetry and cost constraints. At the execution layer, we formalize deliberation as a Macro-Scale Recurrent Neural Network (RNN), where the consensus state loops back through a semantic forget gate to enable iterative refinement without proportional VRAM scaling. Key components include an orchestration fabric for trustless N-to-N peer review, a Quadratic Voting activation function for non-linear consensus, and a feedback-driven state update. Empirical validation on challenging benchmarks (AIME 2025, LiveCodeBench) demonstrates that this topology allows ensembles of small (less than 20B) consumer-grade models to match or exceed the performance of state-of-the-art 100B+ parameter models, establishing a new hardware arbitrage efficiency frontier. Furthermore, testing on the DarkBench safety suite reveals intrinsic alignment properties, with peer-mediated correction reducing sycophancy scores below that of any individual agent.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 31 of its 44 references that are in the map. First posted on arXiv 23 Jan 2026.

Builds on

31 of 44 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
19 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  6. 2018Computational Optimal Transportcited by 765
  7. 2019Green AIcited by 620
  8. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  9. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  10. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  11. 2024Titans: Learning to Memorize at Test Timecited by 303
  12. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  13. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  14. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  15. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  16. 2025End-to-End Test-Time Training for Long Contextcited by 53
  17. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  18. 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 25
  19. 2025From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibriumcited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Emergent Collaborative Deliberation in Multi-Model AI Systems: A BFT-Derived Protocol for Epistemic Synthesis
  2. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  3. 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agents
  4. 2026Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoning
  5. 2026Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensus
  6. 2025Online Federation For Mixtures of Proprietary Agents with Black-Box Encoders
  7. 2026Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling
  8. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms