mlsanityData to 7 Oct 2026

CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents

arXiv 2407.01511 ↗First posted 1 Jul 2024cs.AIcited by 38

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and the complexities of constructing tasks and evaluators. To overcome these limitations, we introduce Crab, the first agent benchmark framework designed to support cross-environment tasks, incorporating a graph-based fine-grained evaluation method and an efficient mechanism for task and evaluator construction. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging Crab, we developed a cross-platform Crab Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated four advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%. All framework code, agent code, and task datasets are publicly available at https://github.com/camel-ai/crab.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 65 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

50 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  2. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  4. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 46

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
34 more
  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  5. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  6. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  7. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  8. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  9. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  10. 2023RoMa: Robust Dense Feature Matchingcited by 296
  11. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  12. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  13. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  14. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  15. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  16. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  17. 2023Planning with Large Language Models for Code Generationcited by 179
  18. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  19. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  20. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  21. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  22. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  23. 2024XFeat: Accelerated Features for Lightweight Image Matchingcited by 113
  24. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  25. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  26. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  27. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  28. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agentscited by 63
  29. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 62
  30. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  31. 2024OmniGlue: Generalizable Feature Matching with Foundation Model Guidancecited by 55
  32. 2023SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Modelscited by 53
  33. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  34. 2024MARE: Multi-Agents Collaboration Framework for Requirements Engineeringcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 17; 2026: 10.

38 papers.

Sort:

By area:

  1. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  5. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  6. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  7. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  8. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
  9. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  10. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  11. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  12. 2025PolySkill: Learning Generalizable Skills Through Polymorphic Abstractioncited by 25
26 more
  1. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  2. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24
  3. 2025R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learningcited by 15
  4. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigationcited by 15
  5. 2024COMMA: A Communicative Multimodal Multi-Agent Benchmarkcited by 12
  6. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  7. 2025CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidancecited by 11
  8. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  9. 2024Agent Skill Acquisition for Large Language Models via CycleQDcited by 10
  10. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 10
  11. 2025PPTArena: A Benchmark for PowerPoint Editingcited by 9
  12. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  13. 2025ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environmentscited by 7
  14. 2025ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  15. 2025ConsumerBench: Benchmarking Generative AI Applications on End-User Devicescited by 2
  16. 2026Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safetycited by 2
  17. 2026What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agentscited by 2
  18. 2026ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaborationcited by 1
  19. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agentscited by 1
  20. 2026ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesiscited by 1
  21. 2025Cybernaut: Towards Reliable Web Automationcited by 0
  22. 2026CORE: Code-based Inverse Self-Training Framework with Graph Expansion for Virtual Agentscited by 0
  23. 2026TVWorld: Foundations for Remote-Control TV Agentscited by 0
  24. 2026Running the Gauntlet: Challenging Agentic Taskscited by 0
  25. 2026OctoNest: Adaptive Cross-Device Execution through Stateful Controlcited by 0
  26. 2026DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
  2. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systems
  3. 2025X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System
  4. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition
  5. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
  6. 2026AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
  7. 2024MMInA: Benchmarking Multihop Multimodal Internet Agents
  8. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments