mlsanityData to 7 Oct 2026

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web

arXiv 2402.17553 ↗First posted 27 Feb 2024cs.AI cs.CL cs.CVcited by 156

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

For decades, human-computer interaction has fundamentally been manual. Even today, almost all productive work done on the computer necessitates human input at every step. Autonomous virtual agents represent an exciting step in automating many of these menial tasks. Virtual agents would empower users with limited technical proficiency to harness the full possibilities of computer systems. They could also enable the efficient streamlining of numerous computer tasks, ranging from calendar management to complex travel bookings, with minimal human intervention. In this paper, we introduce OmniACT, the first-of-a-kind dataset and benchmark for assessing an agent's capability to generate executable programs to accomplish computer tasks. Our scope extends beyond traditional web automation, covering a diverse range of desktop applications. The dataset consists of fundamental tasks such as "Play the next song", as well as longer horizon tasks such as "Send an email to John Doe mentioning the time and place to meet". Specifically, given a pair of screen image and a visually-grounded natural language task, the goal is to generate a script capable of fully executing the task. We run several strong baseline language model agents on our benchmark. The strongest baseline, GPT-4, performs the best on our benchmark However, its performance level still reaches only 15% of the human proficiency in generating executable scripts capable of completing the task, demonstrating the challenge of our task for conventional web agents. Our benchmark provides a platform to measure and evaluate the progress of language model agents in automating computer tasks and motivates future work towards building multimodal models that bridge large language models and the visual grounding of computer screens.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 64 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

43 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 42

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
30 more
  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  5. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  6. 2023Language Models can Solve Computer Taskscited by 371
  7. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  9. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  10. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  11. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  12. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  13. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  14. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  15. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  16. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  17. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  18. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  19. 2022A data-driven approach for learning to control computerscited by 87
  20. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  21. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  22. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  23. 2018Learning to Navigate the Webcited by 60
  24. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  25. 2021Grounding Open-Domain Instructions to Automate Web Support Taskscited by 43
  26. 2023Hierarchical Prompting Assists Large Language Model on Web Navigationcited by 36
  27. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  28. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  29. 2023Becoming self-instruct: introducing early stopping criteria for minimal instruct tuningcited by 13
  30. 2023Lexi: Self-Supervised Learning of the UI Languagecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 66; 2026: 55.

156 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  7. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  8. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  9. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  10. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  12. 2025Survey on Evaluation of LLM-based Agentscited by 190
38 more
  1. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  2. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  3. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  4. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  5. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  6. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  7. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  8. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  9. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  10. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  11. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  12. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  13. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  14. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  15. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  16. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  17. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  18. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  19. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  20. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  21. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  22. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  23. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  24. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  25. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  26. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  27. 2025EvoAgentX: An Automated Framework for Evolving Agentic Workflowscited by 43
  28. 2024Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Modelscited by 41
  29. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  30. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  31. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  32. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  33. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  34. 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agentscited by 25
  35. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  36. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Groundingcited by 20
  37. 2024Infogent: An Agent-Based Framework for Web Information Aggregationcited by 19
  38. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Controlcited by 19

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios
  2. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
  3. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human
  4. 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
  5. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
  6. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
  7. 2026Qwen3.8-Omni: Towards Native Omni-Modal Agents
  8. 2025OmniNova:A General Multimodal Agent Framework