# macOSWorld: A Multilingual Interactive Benchmark for GUI Agents

arXiv: https://arxiv.org/abs/2506.04135 · HTML: https://mlsanity.com/paper/2506.04135/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 44 of its 57 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 44)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control](https://mlsanity.com/paper/2306.07863/) (2023)
- [You Only Look at Screens: Multimodal Chain-of-Action Agents](https://mlsanity.com/paper/2309.11436/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage](https://mlsanity.com/paper/2409.11295/) (2024)
- [UFO: A UI-Focused Agent for Windows OS Interaction](https://mlsanity.com/paper/2402.07939/) (2024)
- [AutoWebGLM: A Large Language Model-based Web Navigating Agent](https://mlsanity.com/paper/2404.03648/) (2024)

## Built on by (most cited first, 30 of 32)

- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks](https://mlsanity.com/paper/2604.24964/) (2026)
- [MMSkills: Towards Multimodal Skills for General Visual Agents](https://mlsanity.com/paper/2605.13527/) (2026)
- [Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond](https://mlsanity.com/paper/2604.22748/) (2026)
- [MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research](https://mlsanity.com/paper/2605.26114/) (2026)
- [TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents](https://mlsanity.com/paper/2606.28480/) (2026)
- [UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents](https://mlsanity.com/paper/2607.04425/) (2026)
- [SCUBA: Salesforce Computer Use Benchmark](https://mlsanity.com/paper/2509.26506/) (2025)
- [OSExpert: Computer-Use Agents Learning Professional Skills via Exploration](https://mlsanity.com/paper/2603.07978/) (2026)
- [iOSWorld: A Benchmark for Personally Intelligent Phone Agents](https://mlsanity.com/paper/2606.09764/) (2026)
- [MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents](https://mlsanity.com/paper/2606.16748/) (2026)
- [VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation](https://mlsanity.com/paper/2604.21375/) (2026)
- [PhoneBuddy: Training Open Models for Agentic Phone Use](https://mlsanity.com/paper/2606.23049/) (2026)
- [GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Tasks](https://mlsanity.com/paper/2510.26098/) (2025)
- [UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability](https://mlsanity.com/paper/2511.17131/) (2025)
- [Video-Based Reward Modeling for Computer-Use Agents](https://mlsanity.com/paper/2603.10178/) (2026)
- [ActionMap: Robot Policy Learning via Voxel Action Heatmap](https://mlsanity.com/paper/2606.06904/) (2026)
- [OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding](https://mlsanity.com/paper/2607.27155/) (2026)
- [A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants](https://mlsanity.com/paper/2510.06223/) (2025)
- [AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark](https://mlsanity.com/paper/2604.24441/) (2026)
- [CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing](https://mlsanity.com/paper/2605.19484/) (2026)
- [Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents](https://mlsanity.com/paper/2605.28775/) (2026)
- [LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control](https://mlsanity.com/paper/2606.16802/) (2026)
- [Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation](https://mlsanity.com/paper/2606.24515/) (2026)
- [The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents](https://mlsanity.com/paper/2608.06663/) (2026)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)
- [RiskWebWorld: A Realistic Interactive Benchmark for GUI Agents in E-commerce Risk Management](https://mlsanity.com/paper/2604.13531/) (2026)
- [Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability](https://mlsanity.com/paper/2605.07110/) (2026)
- [MacArena: Benchmarking Computer Use Agents on an Online macOS Environment](https://mlsanity.com/paper/2606.06560/) (2026)
- [Tactile: Giving Computer-Using Agents Hands and Feet](https://mlsanity.com/paper/2607.14443/) (2026)

## Abstract (arXiv)

Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks and facilitating accessibility, but existing interactive benchmarks are mostly English-only, covering web-use or Windows, Linux, and Android environments, but not macOS. macOS is a major OS with distinctive GUI patterns and exclusive applications. To bridge the gaps, we present macOSWorld, the first comprehensive benchmark for evaluating GUI agents on macOS. macOSWorld features 202 multilingual interactive tasks across 30 applications (28 macOS-exclusive), with task instructions and OS interfaces offered in 5 languages (English, Chinese, Arabic, Japanese, and Russian). As GUI agents are shown to be vulnerable to deception attacks, macOSWorld also includes a dedicated safety benchmarking subset. Our evaluation on six GUI agents reveals a dramatic gap: proprietary computer-use agents lead at above 30% success rate, while open-source lightweight research models lag at below 5%, highlighting the need for macOS domain adaptation. Multilingual benchmarks also expose common weaknesses, especially in Arabic, with a 28.8% average degradation compared to English. Results from safety benchmarking also highlight that deception attacks are more general and demand immediate attention. Project page: https://macos-world.github.io.