# Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

arXiv: https://arxiv.org/abs/2608.27477 · HTML: https://mlsanity.com/paper/2608.27477/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 26 of 26)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses](https://mlsanity.com/paper/2604.25850/) (2026)
- [Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents](https://mlsanity.com/paper/2602.16855/) (2026)
- [MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments](https://mlsanity.com/paper/2512.19432/) (2025)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies](https://mlsanity.com/paper/2605.03596/) (2026)
- [HiconAgent: History Context-aware Policy Optimization for GUI Agents](https://mlsanity.com/paper/2512.01763/) (2025)
- [MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research](https://mlsanity.com/paper/2605.26114/) (2026)
- [AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications](https://mlsanity.com/paper/2605.27761/) (2026)
- [iOSWorld: A Benchmark for Personally Intelligent Phone Agents](https://mlsanity.com/paper/2606.09764/) (2026)
- [PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions](https://mlsanity.com/paper/2606.14832/) (2026)
- [MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment](https://mlsanity.com/paper/2601.20335/) (2026)
- [PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent](https://mlsanity.com/paper/2603.29318/) (2026)
- [A Task-State Representation for Long-Horizon Mobile GUI Agents](https://mlsanity.com/paper/2607.00502/) (2026)

## Abstract (arXiv)

Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks. Existing benchmarks such as AndroidWorld and MobileWorld provide strong foundations for mobile agent evaluation, but their application coverage and task design do not yet fully capture the diversity and complexity of realistic mobile use. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios. GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. We evaluate eight frontier models and find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements. We further conduct controlled ablation studies of agent harness choices, including context retention and explicit state tracking, under a shared environment, model setting, and task taxonomy. Results show that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models. Overall, GMA complements existing benchmarks by expanding application coverage and task complexity, providing a challenging testbed for evaluating mobile agents and studying how harness design supports reliable execution in complex mobile workflows.