mlsanityData to 7 Oct 2026

ScreenAgent: A Vision Language Model-driven Computer Control Agent

arXiv 2402.07945 ↗First posted 9 Feb 2024cs.HC cs.AI cs.CVcited by 101

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Existing Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled directly by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing a variety of daily computer tasks. Finally, we trained a model, ScreenAgent, which achieved computer control capabilities comparable to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code is available at urlhttps://github.com/niuzaisheng/ScreenAgent.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 37 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

29 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
16 more
  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  8. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  9. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  10. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  11. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  12. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  13. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  14. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  15. 2023ProAgent: From Robotic Process Automation to Agentic Process Automationcited by 26
  16. 2016End-to-End Goal-Driven Web Navigationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 47; 2026: 28.

101 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  3. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  4. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  5. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  6. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  7. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  8. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  9. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  10. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  11. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  12. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
38 more
  1. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  2. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  3. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  4. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  5. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  7. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  8. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  9. 2025UFO2: The Desktop AgentOScited by 38
  10. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  11. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  12. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  13. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  14. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Toolscited by 30
  15. 2024MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devicescited by 29
  16. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  17. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  18. 2024Large Action Models: From Inception to Implementationcited by 18
  19. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  20. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learningcited by 17
  21. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  22. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 14
  23. 2025Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agentscited by 14
  24. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  25. 2025Vision-Language Models Can't See the Obviouscited by 13
  26. 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agentscited by 12
  27. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  28. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Datacited by 10
  29. 2024WEPO: Web Element Preference Optimization for LLM-based Web Navigationcited by 10
  30. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  31. 2024GUI Action Narrator: Where and When Did That Action Take Place?cited by 9
  32. 2024Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensorscited by 9
  33. 2025COLA: A Scalable Multi-Agent Framework For Windows UI Task Automationcited by 9
  34. 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agentscited by 9
  35. 2025Fundamentals of Building Autonomous LLM Agentscited by 7
  36. 2025CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planningcited by 6
  37. 2025Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memorycited by 6
  38. 2025STEVE: A Step Verification Pipeline for Computer-use Agent Trainingcited by 5

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  2. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent
  3. 2024Training a Vision Language Model as Smartphone Assistant
  4. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents
  5. 2025Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spaces
  6. 2026Agent-Computer Observation Interfaces Enable Dynamic Computer Use
  7. 2026CLI-Anything: Towards Agent-Native Computer Use
  8. 2025TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents