mlsanityData to 7 Oct 2026

OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

arXiv 2410.23218 ↗First posted 30 Oct 2024cs.CL cs.CV cs.HCcited by 370

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared to their closed-source counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD) scenarios. To facilitate future research in this area, we developed OS-Atlas - a foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling. We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements. This dataset, combined with innovations in model training, provides a solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces. Through extensive evaluation across six benchmarks spanning three different platforms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.

Key facts. Cited by 370 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 42 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

35 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  2. 2024OmniParser for Pure Vision Based GUI Agent1 Aug
  3. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
  5. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 30

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  8. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  10. 2023Cognitive Architectures for Language Agentscited by 422
  11. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  12. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
18 more
  1. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  2. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  3. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  4. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  5. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  6. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  7. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  8. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  9. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  10. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  11. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  12. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 97
  13. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  14. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  15. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  16. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  17. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  18. 2024Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 189; 2026: 174.

370 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  4. 2025Kimi-VL Technical Reportcited by 381
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  7. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  8. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  9. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  10. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  11. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  12. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
38 more
  1. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  2. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  3. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  4. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  5. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2025MiMo-VL Technical Reportcited by 106
  8. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  9. 2024GUI Agents: A Surveycited by 101
  10. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  11. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  12. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  13. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  14. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  15. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  16. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  17. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  18. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  19. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  20. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  21. 2025Detect Anything via Next Point Predictioncited by 66
  22. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  23. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  24. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  25. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  26. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  27. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  28. 2025FineVision: Open Data Is All You Needcited by 51
  29. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  30. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  31. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  32. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  33. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  34. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  35. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 40
  36. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  37. 2026Code as Agent Harnesscited by 40
  38. 2026UI-Venus-1.5 Technical Reportcited by 39

50 most cited of 370 All 370 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
  2. 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
  3. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
  4. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
  5. 2024Aria-UI: Visual Grounding for GUI Instructions
  6. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
  7. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  8. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset