mlsanityData to 7 Oct 2026

Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

arXiv 2412.04454 ↗First posted 5 Dec 2024cs.CLcited by 271

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Automating GUI tasks remains challenging due to reliance on textual representations, platform-specific action spaces, and limited reasoning capabilities. We introduce Aguvis, a unified vision-based framework for autonomous GUI agents that directly operates on screen images, standardizes cross-platform interactions and incorporates structured reasoning via inner monologue. To enable this, we construct Aguvis Data Collection, a large-scale dataset with multimodal grounding and reasoning annotations, and develop a two-stage training pipeline that separates GUI grounding from planning and reasoning. Experiments show that Aguvis achieves state-of-the-art performance across offline and real-world online benchmarks, marking the first fully autonomous vision-based GUI agent that operates without closed-source models. We open-source all datasets, models, and training recipes at https://aguvis-project.github.io to advance future research.

Key facts. Cited by 271 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 55 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

44 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar

Other references · 43

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
31 more
  1. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  2. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  3. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2023Language Models can Solve Computer Taskscited by 371
  6. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  8. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  9. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  10. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  12. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  13. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  14. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  15. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  16. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  17. 2024Tree Search for Language Model Agentscited by 148
  18. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  19. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  20. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  21. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  22. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  23. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  24. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  25. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  26. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  27. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  28. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  29. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  30. 2023A Zero-Shot Language Agent for Computer Control with Structured Reflectioncited by 28
  31. 2022MUG: Interactive Multimodal Grounding on User Interfacescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 153; 2026: 117.

271 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  4. 2025Kimi-VL Technical Reportcited by 381
  5. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  6. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  7. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  8. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  9. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  10. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  11. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  12. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
38 more
  1. 2025MiMo-VL Technical Reportcited by 106
  2. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  3. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  4. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  5. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  6. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  7. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  8. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  9. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  10. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  11. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  12. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  13. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  14. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  15. 2025FineVision: Open Data Is All You Needcited by 51
  16. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  17. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  18. 2025AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agentscited by 49
  19. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  20. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  21. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  22. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  23. 2026UI-Venus-1.5 Technical Reportcited by 39
  24. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  25. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  26. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  27. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  28. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 33
  29. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 32
  30. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  31. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  32. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  33. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  34. 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 28
  35. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  36. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  37. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  38. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24

50 most cited of 271 All 271 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024TinyClick: Single-Turn Agent for Empowering GUI Automation
  2. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  3. 2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
  4. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
  5. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
  6. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
  7. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  8. 2025SpiritSight Agent: Advanced GUI Agent with One Look