# Adaptive Vision-Language Model Routing for Computer Use Agents arXiv: https://arxiv.org/abs/2603.12823 ยท HTML: https://mlsanity.com/paper/2603.12823/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 23 of its 32 references that are in the map. - First posted on arXiv 13 Mar 2026. ## Placed in - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 23 of 23) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025) - [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023) - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023) - [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023) - [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025) - [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024) - [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023) - [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024) - [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025) - [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025) - [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024) - [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025) - [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025) - [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025) - [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023) - [OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents](https://mlsanity.com/paper/2506.14866/) (2025) - [LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordination](https://mlsanity.com/paper/2309.11696/) (2023) - [UFO2: The Desktop AgentOS](https://mlsanity.com/paper/2504.14603/) (2025) - [RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents](https://mlsanity.com/paper/2506.00618/) (2025) - [A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?](https://mlsanity.com/paper/2505.10924/) (2025) - [Measuring Harmfulness of Computer-Using Agents](https://mlsanity.com/paper/2508.00935/) (2025) ## Built on by (most cited first, 8 of 8) - [Agent-as-a-Router: Agentic Model Routing for Coding Tasks](https://mlsanity.com/paper/2606.22902/) (2026) - [Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines](https://mlsanity.com/paper/2604.15376/) (2026) - [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026) - [DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?](https://mlsanity.com/paper/2606.12402/) (2026) - [Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents](https://mlsanity.com/paper/2608.06171/) (2026) - [Scaling Mobile Agent Systems: From Capability Density to Collective Intelligence](https://mlsanity.com/paper/2605.08124/) (2026) - [Uncertainty-Aware Decision Making in Multimodal Large Language Models](https://mlsanity.com/paper/2608.17084/) (2026) - [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026) ## Abstract (arXiv) Computer Use Agents (CUAs) translate natural-language instructions into Graphical User Interface (GUI) actions such as clicks, keystrokes, and scrolls by relying on a Vision-Language Model (VLM) to interpret screenshots and predict grounded tool calls. However, grounding accuracy varies dramatically across VLMs, while current CUA systems typically route every action to a single fixed model regardless of difficulty. We propose Adaptive VLM Routing (AVR), a framework that inserts a lightweight semantic routing layer between the CUA orchestrator and a pool of VLMs. For each tool call, AVR estimates action difficulty from multimodal embeddings, probes a small VLM to measure confidence, and routes the action to the cheapest model whose predicted accuracy satisfies a target reliability threshold. For textitwarm agents with memory of prior UI interactions, retrieved context further narrows the capability gap between small and large models, allowing many actions to be handled without escalation. We formalize routing as a cost--accuracy trade-off, derive a threshold-based policy for model selection, and evaluate AVR using ScreenSpot-Pro grounding data together with the OpenClaw agent routing benchmark. Across these settings, AVR projects inference cost reductions of up to 78% while staying within 2 percentage points of an all-large-model baseline. When combined with the Visual Confused Deputy guardrail, AVR also escalates high-risk actions directly to the strongest available model, unifying efficiency and safety within a single routing framework. Materials are also provided Model, benchmark, and code: https://github.com/vllm-project/semantic-router.