# AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios

arXiv: https://arxiv.org/abs/2602.23166 · HTML: https://mlsanity.com/paper/2602.23166/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 30 of 31)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning](https://mlsanity.com/paper/2505.14362/) (2025)
- [CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs](https://mlsanity.com/paper/2406.18521/) (2024)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning](https://mlsanity.com/paper/2505.08617/) (2025)
- [DeepEyesV2: Toward Agentic Multimodal Model](https://mlsanity.com/paper/2511.05271/) (2025)
- [Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search](https://mlsanity.com/paper/2509.07969/) (2025)
- [WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent](https://mlsanity.com/paper/2508.05748/) (2025)
- [The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution](https://mlsanity.com/paper/2510.25726/) (2025)
- [WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences](https://mlsanity.com/paper/2406.11069/) (2024)
- [m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks](https://mlsanity.com/paper/2403.11085/) (2024)
- [MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents](https://mlsanity.com/paper/2508.21475/) (2025)
- [TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning](https://mlsanity.com/paper/2511.01833/) (2025)
- [Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning](https://mlsanity.com/paper/2510.12712/) (2025)
- [VisionArena: 230K Real World User-VLM Conversations with Preference Labels](https://mlsanity.com/paper/2412.08687/) (2024)
- [Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks](https://mlsanity.com/paper/2505.24876/) (2025)

## Built on by (most cited first, 23 of 23)

- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision](https://mlsanity.com/paper/2606.01139/) (2026)
- [VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining](https://mlsanity.com/paper/2603.15030/) (2026)
- [Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents](https://mlsanity.com/paper/2605.10832/) (2026)
- [Agent Skill Evaluation and Evolution: Frameworks and Benchmarks](https://mlsanity.com/paper/2606.11435/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains](https://mlsanity.com/paper/2606.02357/) (2026)
- [TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents](https://mlsanity.com/paper/2605.09934/) (2026)
- [VistaHop: Benchmarking Long-Horizon Visual DeepSearch](https://mlsanity.com/paper/2606.03273/) (2026)
- [MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents](https://mlsanity.com/paper/2607.11818/) (2026)
- [Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios](https://mlsanity.com/paper/2604.14041/) (2026)
- [Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution](https://mlsanity.com/paper/2605.08887/) (2026)
- [DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment](https://mlsanity.com/paper/2607.07820/) (2026)
- [Self-Evolving Code-with-Image Reasoning](https://mlsanity.com/paper/2608.11292/) (2026)
- [Learning to Learn from Multimodal Experience](https://mlsanity.com/paper/2605.16857/) (2026)
- [Beacon: Knowing When and How to Perform Agentic Visual Reasoning](https://mlsanity.com/paper/2607.28595/) (2026)
- [SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding](https://mlsanity.com/paper/2609.05141/) (2026)
- [Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models](https://mlsanity.com/paper/2606.12969/) (2026)
- [Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents](https://mlsanity.com/paper/2608.25417/) (2026)
- [AtomRec: Evolving Atomic Memory for Agentic Recommendation](https://mlsanity.com/paper/2609.04882/) (2026)
- [DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents](https://mlsanity.com/paper/2609.06059/) (2026)
- [SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agents](https://mlsanity.com/paper/2609.34557/) (2026)
- [V-Gym: Enhancing Agentic Visual Reasoning via Skill-Data Co-Evolution](https://mlsanity.com/paper/2609.34682/) (2026)

## Abstract (arXiv)

Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip by interpreting a transit map and checking schedules under routing constraints. However, existing multimodal benchmarks mainly evaluate single-turn visual reasoning or specific tool skills, and they do not fully capture the realism, visual subtlety, and long-horizon tool use that practical agents require. We introduce AgentVista, a benchmark for generalist multimodal agents that spans 25 sub-domains across 7 categories, pairing realistic and detail-rich visual scenarios with natural hybrid tool use. Tasks require long-horizon tool interactions across modalities, including web search, image search, page navigation, and code-based operations for both image processing and general programming. Comprehensive evaluation of state-of-the-art models exposes significant gaps in their ability to carry out long-horizon multimodal tool use. Even the best model in our evaluation, Gemini-3-Pro with tools, achieves only 27.3% overall accuracy, and hard instances can require more than 25 tool-calling turns. We expect AgentVista to accelerate the development of more capable and reliable multimodal agents for realistic and ultra-challenging problem solving.