mlsanityData to 7 Oct 2026

AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents

arXiv 2407.17490 ↗First posted 3 Jul 2024cs.HC cs.AI cs.MMcited by 110

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents which are capable of completing tasks by directly interacting with the graphical user interface (GUI) on mobile devices. AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels. Unlike existing GUI-related datasets, e.g., Rico, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we finetune a baseline model SPHINX Agent and illustrate the effectiveness of AMEX.The project is available at https://yxchai.com/AMEX/.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 37 of its 50 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

37 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  2. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  3. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation19 Feb
  4. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb
  5. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  6. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  7. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  8. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 29

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2023Mistral 7Bcited by 3,065
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  11. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
17 more
  1. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  3. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  4. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  7. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  8. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  9. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  10. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  11. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  12. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  13. 2021VINS: Visual Search for Mobile User Interface Designcited by 37
  14. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28
  15. 2022UGIF: UI Grounded Instruction Followingcited by 25
  16. 2024MobileAgent: enhancing mobile control via human-machine interaction and SOP integrationcited by 18
  17. 2024MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modelingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 59; 2026: 41.

110 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  4. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  5. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  6. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  7. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  10. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  11. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
38 more
  1. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  2. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  3. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  4. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  5. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  6. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  7. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  8. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  9. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  10. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  11. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  12. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  13. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  14. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  15. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  16. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  17. 2025FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agentscited by 19
  18. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  19. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  20. 2025A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluationcited by 17
  21. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  22. 2025Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agentscited by 14
  23. 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 14
  24. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  25. 2026Generative Visual Code Mobile World Modelscited by 13
  26. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 12
  27. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  28. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  29. 2026UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agentscited by 11
  30. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  31. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10
  32. 2025Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agentcited by 10
  33. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
  34. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  35. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
  36. 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 10
  37. 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learningcited by 9
  38. 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selectioncited by 9

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  2. 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  3. 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild
  4. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
  5. 2026Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?
  6. 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization
  7. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation
  8. 2023Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing