# Why Keep Your Doubts to Yourself? Trading Visual Uncertainties in Multi-Agent Bandit Systems

arXiv: https://arxiv.org/abs/2601.18735 · HTML: https://mlsanity.com/paper/2601.18735/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy](https://mlsanity.com/paper/2412.02210/) (2024)
- [KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems](https://mlsanity.com/paper/2502.07350/) (2025)
- [Contextual Bandits with Large Action Spaces: Made Practical](https://mlsanity.com/paper/2207.05836/) (2022)
- [Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems](https://mlsanity.com/paper/2403.02419/) (2024)
- [Variance-Aware Sparse Linear Bandits](https://mlsanity.com/paper/2205.13450/) (2022)
- [Large Language Model-Enhanced Multi-Armed Bandits](https://mlsanity.com/paper/2502.01118/) (2025)
- [Causal Graphical Models for Vision-Language Compositional Understanding](https://mlsanity.com/paper/2412.09353/) (2024)
- [MIX-MAB: Reinforcement Learning-based Resource Allocation Algorithm for LoRaWAN](https://mlsanity.com/paper/2206.03401/) (2022)
- [Beyond Numeric Rewards: In-Context Dueling Bandits with LLM Agents](https://mlsanity.com/paper/2407.01887/) (2024)

## Built on by (most cited first, 4 of 4)

- [AgriWorld:A World Tools Protocol Framework for Verifiable Agricultural Reasoning with Code-Executing LLM Agents](https://mlsanity.com/paper/2602.15325/) (2026)
- [LogicGaze: Benchmarking Causal Consistency in Visual Narratives via Counterfactual Verification](https://mlsanity.com/paper/2602.00292/) (2026)
- [Bridging the Semantic Chasm: Synergistic Conceptual Anchoring for Generalized Few-Shot and Zero-Shot OOD Perception](https://mlsanity.com/paper/2602.00340/) (2026)
- [CacheScope: Exact-Compute Allocation for Training-Free Diffusion Acceleration](https://mlsanity.com/paper/2610.12530/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination. We introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3x. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems.