mlsanityData to 14 Oct 2026

GUICourse: From General Vision Language Models to Versatile GUI Agents

arXiv 2406.11317 ↗First posted 17 Jun 2024cs.AI cs.CL cs.CVcited by 114

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Utilizing Graphic User Interface (GUI) for human-computer interaction is essential for accessing a wide range of digital tools. Recent advancements in Vision Language Models (VLMs) highlight the compelling potential to develop versatile agents to help humans finish GUI navigation tasks. However, current VLMs are challenged in terms of fundamental abilities (OCR and grounding) and GUI knowledge (the functions and control methods of GUI elements), preventing them from becoming practical GUI agents. To solve these challenges, we contribute GUICourse, a suite of datasets to train visual-based GUI agents from general VLMs. First, we introduce the GUIEnv dataset to strengthen the OCR and grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat datasets to enrich their knowledge of GUI components and interactions. Experiments demonstrate that our GUI agents have better performance on common GUI tasks than their baseline VLMs. Even the small-size GUI agent (with 3.1B parameters) can still work well on single-step and multi-step GUI tasks. Finally, we analyze the different varieties in the training stage of this agent by ablation study. Our source codes and datasets are released at https://github.com/yiye3/GUICourse.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 72 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

50 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understanding7 Feb
  2. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  3. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2019Towards VQA Models That Can Readcited by 1,718
35 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  6. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  7. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  9. 2023Language Models can Solve Computer Taskscited by 372
  10. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  12. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  13. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  14. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  15. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  16. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 196
  17. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  18. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  19. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  20. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  21. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  22. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  23. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  24. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  25. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  26. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  27. 2022A data-driven approach for learning to control computerscited by 88
  28. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  29. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  30. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 68
  31. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  32. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  33. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  34. 2021Grounding Open-Domain Instructions to Automate Web Support Taskscited by 43
  35. 2024Exploring Perceptual Limitation of Multimodal Large Language Modelscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 63; 2026: 36.

114 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  2. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  4. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  5. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  6. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
  7. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  8. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  9. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  10. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  11. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  12. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
38 more
  1. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  4. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  5. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  6. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  7. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  8. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  9. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  10. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  11. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  12. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 42
  13. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  14. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 38
  15. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  16. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 34
  17. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  18. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 28
  19. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  20. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  21. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 25
  22. 2026Reinforcement World Model Learning for LLM-based Agentscited by 24
  23. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  24. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  25. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 23
  26. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  27. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  28. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understandingcited by 19
  29. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  30. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  31. 2024Large Action Models: From Inception to Implementationcited by 18
  32. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
  33. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  34. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  35. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
  36. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
  37. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 14
  38. 2024Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuningcited by 14

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
  2. 2026Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
  3. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
  4. 2025TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents
  5. 2025GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Tasks
  6. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  7. 2025Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation
  8. 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents