mlsanityData to 7 Oct 2026

SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

arXiv 2401.10935 ↗First posted 17 Jan 2024cs.HC cs.AIcited by 553

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a novel visual GUI agent -- SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding -- the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code are available at https://github.com/njucckevin/SeeClick.

Key facts. Cited by 553 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 57 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

40 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  2. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 38

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
26 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  2. 2023Language Models can Solve Computer Taskscited by 371
  3. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  4. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  5. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  6. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  7. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  8. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  9. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  10. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  11. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  12. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  13. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  14. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  15. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  16. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  17. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  18. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  19. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  20. 2018Learning to Navigate the Webcited by 60
  21. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  22. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  23. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38
  24. 2020Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?cited by 29
  25. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  26. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 77; 2025: 237; 2026: 239.

553 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  7. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  10. 2025Kimi-VL Technical Reportcited by 381
  11. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  12. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
38 more
  1. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  2. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  4. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  6. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  7. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  8. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  9. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  10. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  12. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  13. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  14. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  15. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  16. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  17. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  18. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  19. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  20. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  21. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  22. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  23. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  24. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  25. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  26. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  27. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  28. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  29. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  30. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  31. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  32. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  33. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  34. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  35. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  36. 2025MiMo-VL Technical Reportcited by 106
  37. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  38. 2024GUI Agents: A Surveycited by 101

50 most cited of 553 All 553 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  2. 2025ProBench: Benchmarking GUI Agents with Accurate Process Information
  3. 2026SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis
  4. 2026Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
  5. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
  6. 2026Recurrent Latent Visual Search for GUI Grounding
  7. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents
  8. 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback