mlsanityData to 14 Oct 2026

The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use

arXiv 2411.10323 ↗First posted 15 Nov 2024cs.AI cs.CL cs.CVcited by 63

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

The recently released model, Claude 3.5 Computer Use, stands out as the first frontier AI model to offer computer use in public beta as a graphical user interface (GUI) agent. As an early beta, its capability in the real-world complex environment remains unknown. In this case study to explore Claude 3.5 Computer Use, we curate and organize a collection of carefully designed tasks spanning a variety of domains and software. Observations from these cases demonstrate Claude 3.5 Computer Use's unprecedented ability in end-to-end language to desktop actions. Along with this study, we provide an out-of-the-box agent framework for deploying API-based GUI automation models with easy implementation. Our case studies aim to showcase a groundwork of capabilities and limitations of Claude 3.5 Computer Use with detailed analyses and bring to the fore questions about planning, action, and critic, which must be considered for future improvement. We hope this preliminary exploration will inspire future research into the GUI agent community. All the test cases in the paper can be tried through the project: https://github.com/showlab/computer_use_ootb.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 39 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

29 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  2. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb
  3. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  5. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  6. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  7. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
10 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  2. 2023Cognitive Architectures for Language Agentscited by 429
  3. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  4. 2024Tree Search for Language Model Agentscited by 151
  5. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  6. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  7. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  8. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  9. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  10. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 42; 2026: 17.

63 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  2. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  3. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  4. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  5. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  6. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  7. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 59
  8. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  9. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  10. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 45
  11. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  12. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 34
38 more
  1. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 32
  2. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
  3. 2025FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spacescited by 28
  4. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agentscited by 28
  5. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 27
  6. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  7. 2025What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commercecited by 22
  8. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Modelscited by 21
  9. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 20
  10. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 19
  11. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learningcited by 18
  12. 2025ScaleTrack: Scaling and back-tracking Automated GUI Agentscited by 16
  13. 2025Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversightcited by 16
  14. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 14
  15. 2024PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital Worldcited by 12
  16. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 12
  17. 2025Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agentcited by 11
  18. 2025In-Context Defense in Computer Agents: An Empirical Studycited by 10
  19. 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selectioncited by 10
  20. 2025Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool usecited by 10
  21. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approachcited by 8
  22. 2025sudo rm -rf agentic_securitycited by 8
  23. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  24. 2025Measuring Harmfulness of Computer-Using Agentscited by 8
  25. 2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Groundingcited by 8
  26. 2026ShowUI-Aloha: Human-Taught GUI Agentcited by 8
  27. 2025Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsingcited by 7
  28. 2025ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environmentscited by 7
  29. 2025OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?cited by 6
  30. 2025When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Taskscited by 6
  31. 2026Enhancing Web Agents with a Hierarchical Memory Treecited by 6
  32. 2026DRS-GUI: Dynamic Region Search for Training-Free GUI Groundingcited by 5
  33. 2024StarWhisper Telescope: An AI framework for automating end-to-end astronomical observationscited by 4
  34. 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worldscited by 4
  35. 2025Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agentcited by 4
  36. 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Groundingcited by 4
  37. 2025FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Gamescited by 3
  38. 2026BAMI: Training-Free Bias Mitigation in GUI Groundingcited by 3

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GUI Agents: A Survey
  2. 2026UI-Venus-1.5 Technical Report
  3. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
  4. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents
  5. 2024GUI Agents with Foundation Models: A Comprehensive Survey
  6. 2025Computer-Use Agents as Judges for Generative User Interface
  7. 2025ProBench: Benchmarking GUI Agents with Accurate Process Information
  8. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents