mlsanityData to 14 Oct 2026

Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

arXiv 2506.08972 ↗First posted 10 Jun 2025cs.CLcited by 12

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Autonomous agents powered by multimodal large language models have been developed to facilitate task execution on mobile devices. However, prior work has predominantly focused on atomic tasks -- such as shot-chain execution tasks and single-screen grounding tasks -- while overlooking the generalization to compositional tasks, which are indispensable for real-world applications. This work introduces UI-NEXUS, a comprehensive benchmark designed to evaluate mobile agents on three categories of compositional operations: Simple Concatenation, Context Transition, and Deep Dive. UI-NEXUS supports interactive evaluation in 20 fully controllable local utility app environments, as well as 30 online Chinese and English service apps. It comprises 100 interactive task templates with an average optimal step count of 14.05. Experimental results across a range of mobile agents with agentic workflow or agent-as-a-model show that UI-NEXUS presents significant challenges. Specifically, existing agents generally struggle to balance performance and efficiency, exhibiting representative failure modes such as under-execution, over-execution, and attention drift, causing visible atomic-to-compositional generalization gap. Inspired by these findings, we propose AGENT-NEXUS, a lightweight and efficient scheduling system to tackle compositional mobile tasks. AGENT-NEXUS extrapolates the abilities of existing mobile agents by dynamically decomposing long-horizon tasks to a series of self-contained atomic subtasks. AGENT-NEXUS achieves 24% to 40% task success rate improvement for existing mobile agents on compositional operation tasks within the UI-NEXUS benchmark without significantly sacrificing inference overhead. The demo video, dataset, and code are available on the project page at https://ui-nexus.github.io.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 65 of its 69 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

65 of 69 references.

Closest ancestors · 30

Same direction, newest first

  1. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects28 Apr
  2. 2025SpiritSight Agent: Advanced GUI Agent with One Look5 Mar
  3. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Users4 Mar
  4. 2025PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC20 Feb
  5. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  6. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
  7. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
  8. 2024GUI Agents: A Survey18 Dec
  9. 2024Large Language Model-Brained GUI Agents: A Survey27 Nov
  10. 2024GUI Agents with Foundation Models: A Comprehensive Survey7 Nov
  11. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
  12. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
18 more
  1. 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
  2. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation19 Oct
  3. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents18 Oct
  4. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  6. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  7. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactions5 Aug
  8. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions5 Aug
  9. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents1 Jul
  10. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  11. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents12 Jun
  12. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
  13. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  14. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  15. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  16. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  17. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  18. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 35

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  7. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  9. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2024Agent Workflow Memorycited by 383
  12. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
23 more
  1. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  2. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  3. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  4. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  6. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  7. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  8. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  9. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  10. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  11. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  12. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 163
  13. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  14. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  15. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  16. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
  17. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  18. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  19. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  20. 2025Atom of Thoughts for Markov LLM Test-Time Scalingcited by 54
  21. 2023Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Webcited by 24
  22. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
  23. 2024Generalization of Compositional Tasks with Logical Specification via Implicit Planningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  2. 2026Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?
  3. 2025AppCopilot: Toward General, Accurate, Long-Horizon, and Efficient Mobile Agent
  4. 2026Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
  5. 2026MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment
  6. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
  7. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
  8. 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones