# FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Data

arXiv: https://arxiv.org/abs/2503.05143 · HTML: https://mlsanity.com/paper/2503.05143/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Agentic AI for wireless networks, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Placed in

- LLM agents and tool use / [Agentic AI for wireless networks](https://mlsanity.com/agents/agentic-ai-for-wireless-networks/)

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning](https://mlsanity.com/paper/2406.11896/) (2024)
- [OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://mlsanity.com/paper/2412.19723/) (2024)
- [Android in the Zoo: Chain-of-Action-Thought for GUI Agents](https://mlsanity.com/paper/2403.02713/) (2024)
- [FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning](https://mlsanity.com/paper/2309.00363/) (2023)
- [Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2106.06047/) (2021)
- [AutoGLM: Autonomous Foundation Agents for GUIs](https://mlsanity.com/paper/2411.00820/) (2024)
- [FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model](https://mlsanity.com/paper/2406.17706/) (2024)
- [DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents](https://mlsanity.com/paper/2410.14803/) (2024)
- [MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents](https://mlsanity.com/paper/2406.08184/) (2024)
- [AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning](https://mlsanity.com/paper/2402.15506/) (2024)
- [MiniVLM: A Smaller and Faster Vision-Language Model](https://mlsanity.com/paper/2012.06946/) (2020)
- [Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms](https://mlsanity.com/paper/2410.18967/) (2024)
- [MobileFlow: A Multimodal LLM For Mobile GUI Agent](https://mlsanity.com/paper/2407.04346/) (2024)
- [FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models](https://mlsanity.com/paper/2406.04845/) (2024)
- [AppVLM: A Lightweight Vision Language Model for Online App Control](https://mlsanity.com/paper/2502.06395/) (2025)
- [KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server](https://mlsanity.com/paper/2410.05725/) (2024)
- [MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users](https://mlsanity.com/paper/2502.02982/) (2025)
- [The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models](https://mlsanity.com/paper/2408.01228/) (2024)
- [Training a Vision Language Model as Smartphone Assistant](https://mlsanity.com/paper/2404.08755/) (2024)
- [Dordis: Efficient Federated Learning with Dropout-Resilient Differential Privacy](https://mlsanity.com/paper/2209.12528/) (2022)

## Built on by (most cited first, 3 of 3)

- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [WebPII: Benchmarking Visual PII Detection for Computer-Use Agents](https://mlsanity.com/paper/2603.17357/) (2026)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)

## Abstract (arXiv)

Mobile agents have attracted tremendous research participation recently. Traditional approaches to mobile agent training rely on centralized data collection, leading to high cost and limited scalability. Distributed training utilizing federated learning offers an alternative by harnessing real-world user data, providing scalability and reducing costs. However, pivotal challenges, including the absence of standardized benchmarks, hinder progress in this field. To tackle the challenges, we introduce FedMABench, the first benchmark for federated training and evaluation of mobile agents, specifically designed for heterogeneous scenarios. FedMABench features 6 datasets with 30+ subsets, 8 federated algorithms, 10+ base models, and over 800 apps across 5 categories, providing a comprehensive framework for evaluating mobile agents across diverse environments. Through extensive experiments, we uncover several key insights: federated algorithms consistently outperform local training; the distribution of specific apps plays a crucial role in heterogeneity; and, even apps from distinct categories can exhibit correlations during training. FedMABench is publicly available at: https://github.com/wwh0411/FedMABench with the datasets at: https://huggingface.co/datasets/wwh0411/FedMABench.