mlsanityData to 14 Oct 2026

AgentStudio: A Toolkit for Building General Virtual Agents

arXiv 2403.17918 ↗First posted 26 Mar 2024cs.AIcited by 45

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

General virtual agents need to handle multimodal observations, master complex action spaces, and self-improve in dynamic, open-domain environments. However, existing environments are often domain-specific and require complex setups, which limits agent development and evaluation in real-world settings. As a result, current evaluations lack in-depth analyses that decompose fundamental agent capabilities. We introduce AgentStudio, a trinity of environments, tools, and benchmarks to address these issues. AgentStudio provides a lightweight, interactive environment with highly generic observation and action spaces, e.g., video observations and GUI/API actions. It integrates tools for creating online benchmark tasks, annotating GUI elements, and labeling actions in videos. Based on our environment and tools, we curate an online task suite that benchmarks both GUI interactions and function calling with efficient auto-evaluation. We also reorganize existing datasets and collect new ones using our tools to establish three datasets: GroundUI, IDMBench, and CriticBench. These datasets evaluate fundamental agent abilities, including GUI grounding, learning from videos, and success detection, pointing to the desiderata for robust, general, and open-ended virtual agents.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

42 of 53 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
30 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  7. 2023Language Models can Solve Computer Taskscited by 372
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  10. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  11. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  12. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
  13. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  14. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  15. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  16. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  17. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  18. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 137
  19. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  20. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  21. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  22. 2022A data-driven approach for learning to control computerscited by 88
  23. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 71
  24. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  25. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  26. 2018Learning to Navigate the Webcited by 60
  27. 2023SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Modelscited by 57
  28. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  29. 2023PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completioncited by 30
  30. 2019DOM-Q-NET: Grounded RL on Structured Languagecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 25; 2026: 10.

45 papers.

Sort:

By area:

  1. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  4. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  5. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  6. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  7. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  8. 2025FineVision: Open Data Is All You Needcited by 53
  9. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  10. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  11. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  12. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 38
33 more
  1. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  2. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  3. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 34
  4. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  5. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  6. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  7. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 20
  8. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  9. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  10. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  11. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  12. 2025Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agentcited by 11
  13. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  14. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  15. 2024GUI Action Narrator: Where and When Did That Action Take Place?cited by 9
  16. 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agentscited by 9
  17. 2025Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigationcited by 8
  18. 2025Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediariescited by 8
  19. 2026ClawEnvKit: Automatic Environment Generation for Claw-Like Agentscited by 8
  20. 2024Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refiningcited by 7
  21. 2025AUTO-Explorer: Automated Data Collection for GUI Agentcited by 5
  22. 2025DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agentscited by 4
  23. 2026A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agentscited by 3
  24. 2026Interactive Reward Agent: GUI Task Evaluation via Environment-State Verificationcited by 3
  25. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  26. 2025Automotive-ENV: Benchmarking Multimodal Agents in Vehicle Interface Systemscited by 2
  27. 2026Agent Skills Should Go Beyond Text: The Case for Visual Skillscited by 2
  28. 2026Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Makingcited by 2
  29. 2025Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UIcited by 1
  30. 2026Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agentscited by 1
  31. 2026Software Engineering for and with GUI Agentcited by 1
  32. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  33. 2026Learning to Adapt: Self-Improving Web Agent via Cognitive-Aware Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ActionStudio: A Lightweight Framework for Data and Training of Large Action Models
  2. 2026NitroGen: An Open Foundation Model for Generalist Gaming Agents
  3. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
  4. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  5. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
  6. 2026VirtualEnv: A Platform for Embodied AI Research
  7. 2026Running the Gauntlet: Challenging Agentic Tasks
  8. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials