mlsanityData to 13 Oct 2026

One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries

arXiv 2603.11545 ↗First posted 12 Mar 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

We present an agentic AI framework for autonomous multimodal query processing that coordinates specialized tools across text, image, audio, video, and document modalities. A central Supervisor dynamically decomposes user queries, delegates subtasks to modality-appropriate tools (e.g., object detection, OCR, speech transcription), and synthesizes results through adaptive routing strategies rather than predetermined decision trees. For text-only queries, the framework uses learned routing via RouteLLM, while non-text paths use SLM-assisted modality decomposition. Evaluated on 2,847 queries across 15 task categories, our framework achieves 72% reduction in time-to-accurate-answer, 85% reduction in conversational rework, and 67% cost reduction compared to the matched hierarchical baseline while maintaining accuracy parity. These results demonstrate that intelligent centralized orchestration fundamentally improves multimodal AI deployment economics.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 30 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

27 of 30 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  8. 2023Visual Instruction Tuningcited by 7,827
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,840
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,117
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  12. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,945
15 more
  1. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,354
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,180
  3. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 857
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 815
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 746
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 652
  7. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 362
  8. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
  9. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 154
  10. 2024Large Multimodal Agents: A Surveycited by 104
  11. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  12. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 46
  13. 2023A quantitative study of NLP approaches to question difficulty estimationcited by 12
  14. 2025One Head, Many Models: Cross-Attention Routing for Cost-Aware LLM Selectioncited by 4
  15. 2025Towards Resource-Efficient Multimodal Intelligence: Learned Routing among Specialized Expert Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation
  2. 2025AgentMaster: A Multi-Agent Conversational Framework Using A2A and MCP Protocols for Multimodal Information Retrieval and Analysis
  3. 2025Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
  4. 2026Modality-Native Routing in Agent-to-Agent Networks: A Multimodal A2A Protocol Extension
  5. 2025STRIDE: A Systematic Framework for Selecting AI Modalities -- Agentic AI, AI Assistants, or LLM Calls
  6. 2025DeepEyesV2: Toward Agentic Multimodal Model
  7. 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?
  8. 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search