# A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation

arXiv: https://arxiv.org/abs/2501.01149 · HTML: https://mlsanity.com/paper/2501.01149/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 8 of 8)

- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://mlsanity.com/paper/2407.17490/) (2024)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation](https://mlsanity.com/paper/2410.15164/) (2024)
- [Understanding the Weakness of Large Language Model Agents within a Complex Android Environment](https://mlsanity.com/paper/2402.06596/) (2024)
- [Benchmarking Mobile Device Control Agents across Diverse Configurations](https://mlsanity.com/paper/2404.16660/) (2024)

## Built on by (most cited first, 19 of 19)

- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [ProBench: Benchmarking GUI Agents with Accurate Process Information](https://mlsanity.com/paper/2511.09157/) (2025)
- [MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research](https://mlsanity.com/paper/2605.26114/) (2026)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild](https://mlsanity.com/paper/2602.11750/) (2026)
- [MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization](https://mlsanity.com/paper/2606.19930/) (2026)
- [TextAtari: 100K Frames Game Playing with Language Agents](https://mlsanity.com/paper/2506.04098/) (2025)
- [UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents](https://mlsanity.com/paper/2605.29534/) (2026)
- [Do Phone-Use Agents Respect Your Privacy?](https://mlsanity.com/paper/2604.00986/) (2026)
- [Jev-Mobile: Jev as an Executor for Mobile GUI Agents](https://mlsanity.com/paper/2609.30186/) (2026)
- [Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decomposition](https://mlsanity.com/paper/2602.07787/) (2026)
- [ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis](https://mlsanity.com/paper/2605.25160/) (2026)
- [PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents](https://mlsanity.com/paper/2605.29486/) (2026)
- [CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents](https://mlsanity.com/paper/2606.12666/) (2026)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)
- [ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults](https://mlsanity.com/paper/2609.04850/) (2026)
- [Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?](https://mlsanity.com/paper/2609.34139/) (2026)
- [OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools](https://mlsanity.com/paper/2610.14157/) (2026)
- [How GUI Agents Fail: GraceBench for Execution Stability Diagnosis](https://mlsanity.com/paper/2610.19900/) (2026)

## Abstract (arXiv)

The advancement of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has catalyzed the development of mobile graphic user interface (GUI) AI agents, which is designed to autonomously perform tasks on mobile devices. However, a significant gap persists in mobile GUI agent evaluation, where existing benchmarks predominantly rely on either static frame assessments such as AndroidControl or offline static apps such as AndroidWorld and thus fail to capture agent performance in dynamic, real-world online mobile apps. To address this gap, we present Android Agent Arena (A3), a novel "essential-state" based procedural evaluation system for mobile GUI agents. A3 introduces a benchmark of 100 tasks derived from 20 widely-used, dynamic online apps across 20 categories from the Google Play Store, ensuring evaluation comprehension. A3 also presents a novel "essential-state" based procedural evaluation method that leverages MLLMs as reward models to progressively verify task completion and process achievement. This evaluation approach address the limitations of traditional function based evaluation methods on online dynamic apps. Furthermore, A3 includes a toolkit to streamline Android device interaction, reset online environment and apps and facilitate data collection from both human and agent demonstrations. The complete A3 system, including the benchmark and tools, will be publicly released to provide a robust foundation for future research and development in mobile GUI agents.