# MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions

arXiv: https://arxiv.org/abs/2502.16796 · HTML: https://mlsanity.com/paper/2502.16796/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 65 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 30 of 46)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration](https://mlsanity.com/paper/2406.01014/) (2024)
- [You Only Look at Screens: Multimodal Chain-of-Action Agents](https://mlsanity.com/paper/2309.11436/) (2023)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Android in the Zoo: Chain-of-Action-Thought for GUI Agents](https://mlsanity.com/paper/2403.02713/) (2024)
- [GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation](https://mlsanity.com/paper/2311.07562/) (2023)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://mlsanity.com/paper/2010.04295/) (2020)
- [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://mlsanity.com/paper/2107.13731/) (2021)
- [On Generative Agents in Recommendation](https://mlsanity.com/paper/2310.10108/) (2023)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments](https://mlsanity.com/paper/2403.11807/) (2024)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents](https://mlsanity.com/paper/2406.08184/) (2024)
- [Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus](https://mlsanity.com/paper/2209.14927/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels](https://mlsanity.com/paper/2101.04893/) (2021)

## Built on by (most cited first, 16 of 16)

- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [The Truth Becomes Clearer Through Debate! Multi-Agent Systems with Large Language Models Unmask Fake News](https://mlsanity.com/paper/2505.08532/) (2025)
- [Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Training](https://mlsanity.com/paper/2506.20332/) (2025)
- [Agent-SAMA: State-Aware Mobile Assistant](https://mlsanity.com/paper/2505.23596/) (2025)
- [STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization](https://mlsanity.com/paper/2511.13091/) (2025)
- [MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation](https://mlsanity.com/paper/2507.16853/) (2025)
- [How Mobile World Model Guides GUI Agents?](https://mlsanity.com/paper/2605.10347/) (2026)
- [AppAgent-Pro: A Proactive GUI Agent System for Multidomain Information Integration and User Assistance](https://mlsanity.com/paper/2508.18689/) (2025)
- [InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2508.19679/) (2025)
- [AgenticGEO: A Self-Evolving Agentic System for Generative Engine Optimization](https://mlsanity.com/paper/2603.20213/) (2026)
- [DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces](https://mlsanity.com/paper/2509.13444/) (2025)
- [PhoneBuddy: Training Open Models for Agentic Phone Use](https://mlsanity.com/paper/2606.23049/) (2026)
- [PPAI: Enabling Personalized LLM Agent Interoperability for Collaborative Edge Intelligence](https://mlsanity.com/paper/2605.18067/) (2026)
- [PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents](https://mlsanity.com/paper/2605.29486/) (2026)
- [Compass: Navigating Global Marine Lead Data Integration through Expert-Guided LLM Agent](https://mlsanity.com/paper/2605.29966/) (2026)

## Abstract (arXiv)

Mobile phone agents can assist people in automating daily tasks on their phones, which have emerged as a pivotal research spotlight. However, existing procedure-oriented agents struggle with cross-app instructions, due to the following challenges: (1) complex task relationships, (2) diverse app environment, and (3) error propagation and information loss in multi-step execution. Drawing inspiration from object-oriented programming principles, we recognize that object-oriented solutions is more suitable for cross-app instruction. To address these challenges, we propose a self-evolving multi-agent framework named MobileSteward, which integrates multiple app-oriented StaffAgents coordinated by a centralized StewardAgent. We design three specialized modules in MobileSteward: (1) Dynamic Recruitment generates a scheduling graph guided by information flow to explicitly associate tasks among apps. (2) Assigned Execution assigns the task to app-oriented StaffAgents, each equipped with app-specialized expertise to address the diversity between apps. (3) Adjusted Evaluation conducts evaluation to provide reflection tips or deliver key information, which alleviates error propagation and information loss during multi-step execution. To continuously improve the performance of MobileSteward, we develop a Memory-based Self-evolution mechanism, which summarizes the experience from successful execution, to improve the performance of MobileSteward. We establish the first English Cross-APP Benchmark (CAPBench) in the real-world environment to evaluate the agents' capabilities of solving complex cross-app instructions. Experimental results demonstrate that MobileSteward achieves the best performance compared to both single-agent and multi-agent frameworks, highlighting the superiority of MobileSteward in better handling user instructions with diverse complexity.