# APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

arXiv: https://arxiv.org/abs/2609.07712 · HTML: https://mlsanity.com/paper/2609.07712/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 24 of 24)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025)
- [AutoWebGLM: A Large Language Model-based Web Navigating Agent](https://mlsanity.com/paper/2404.03648/) (2024)
- [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024)
- [MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents](https://mlsanity.com/paper/2406.08184/) (2024)
- [Understanding the Weakness of Large Language Model Agents within a Complex Android Environment](https://mlsanity.com/paper/2402.06596/) (2024)
- [Benchmarking Mobile Device Control Agents across Diverse Configurations](https://mlsanity.com/paper/2404.16660/) (2024)
- [MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding](https://mlsanity.com/paper/2409.14818/) (2024)
- [ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation](https://mlsanity.com/paper/2312.13108/) (2023)
- [Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment](https://mlsanity.com/paper/2503.15937/) (2025)
- [Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction](https://mlsanity.com/paper/2305.08144/) (2023)
- [LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation](https://mlsanity.com/paper/2404.16054/) (2024)
- [MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research](https://mlsanity.com/paper/2605.26114/) (2026)
- [ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis](https://mlsanity.com/paper/2605.25160/) (2026)

## Abstract (arXiv)

Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.