mlsanityData to 14 Oct 2026

FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems

arXiv 2604.14956 ↗First posted 16 Apr 2026cs.MAcited by 0

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Training GUI agents with traditional centralized methods faces significant cost and scalability challenges. Federated learning (FL) offers a promising solution, yet its potential is hindered by the lack of benchmarks that capture real-world, cross-platform heterogeneity. To bridge this gap, we introduce FedGUI, the first comprehensive benchmark for developing and evaluating federated GUI agents across mobile, web, and desktop platforms. FedGUI provides a suite of six curated datasets to systematically study four crucial types of heterogeneity: cross-platform, cross-device, cross-OS, and cross-source. Extensive experiments reveal several key insights: First, we show that cross-platform collaboration improves performance, extending prior mobile-only federated learning to diverse GUI environments; Second, we demonstrate the presence of distinct heterogeneity dimensions and identify platform and OS as the most influential factors. FedGUI provides a vital foundation for the community to build more scalable and privacy-preserving GUI agents for real-world deployment. Our code and data are publicly available at https://github.com/wwh0411/FedGUI..

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 46 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

34 of 46 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction7 Apr
  2. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
  3. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
  4. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents8 Sep
  5. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
  6. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding29 Jul
  7. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  8. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents3 Jul
  9. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  11. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb

Other references · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  9. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  10. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  11. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  12. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
11 more
  1. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  2. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  3. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  4. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  5. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 58
  6. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 46
  7. 2026Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systemscited by 20
  8. 2026MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environmentscited by 18
  9. 2025MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Toolscited by 11
  10. 2024Identifying User Goals from UI Trajectoriescited by 9
  11. 2022Dordis: Efficient Federated Learning with Dropout-Resilient Differential Privacycited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
  2. 2025FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Data
  3. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users
  4. 2026FedLTLib: A Comprehensive Benchmark for Federated Long-Tail Learning
  5. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
  6. 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
  7. 2025Federated Learning: A Survey on Privacy-Preserving Collaborative Intelligence
  8. 2025Mobile GUI Agents under Real-world Threats: Are We There Yet?