# TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments

arXiv: https://arxiv.org/abs/2505.17629 · HTML: https://mlsanity.com/paper/2505.17629/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 31)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration](https://mlsanity.com/paper/2406.01014/) (2024)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [Aria-UI: Visual Grounding for GUI Instructions](https://mlsanity.com/paper/2412.16256/) (2024)
- [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://mlsanity.com/paper/2410.05993/) (2024)
- [MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices](https://mlsanity.com/paper/2312.16886/) (2023)
- [GUICourse: From General Vision Language Models to Versatile GUI Agents](https://mlsanity.com/paper/2406.11317/) (2024)
- [GUI Agents with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2411.04890/) (2024)
- [Towards Conversational Recommendation over Multi-Type Dialogs](https://mlsanity.com/paper/2005.03954/) (2020)
- [VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents](https://mlsanity.com/paper/2408.06327/) (2024)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://mlsanity.com/paper/2107.13731/) (2021)
- [Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios](https://mlsanity.com/paper/2401.17167/) (2024)
- [CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation](https://mlsanity.com/paper/2402.11941/) (2024)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding](https://mlsanity.com/paper/2409.14818/) (2024)
- [MidMed: Towards Mixed-Type Dialogues for Medical Consultation](https://mlsanity.com/paper/2306.02923/) (2023)
- [E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion](https://mlsanity.com/paper/2406.14250/) (2024)

## Built on by (most cited first, 8 of 8)

- [PRIM: Towards Practical In-Image Multilingual Machine Translation](https://mlsanity.com/paper/2509.05146/) (2025)
- [PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent](https://mlsanity.com/paper/2603.29318/) (2026)
- [DocOS: Towards Proactive Document-Guided Actions in GUI Agents](https://mlsanity.com/paper/2605.18048/) (2026)
- [CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception](https://mlsanity.com/paper/2608.08392/) (2026)
- [Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion](https://mlsanity.com/paper/2602.06351/) (2026)
- [JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition](https://mlsanity.com/paper/2609.10451/) (2026)
- [Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?](https://mlsanity.com/paper/2609.34139/) (2026)
- [Learning Reliable GUI Agents under Imperfect Priors](https://mlsanity.com/paper/2609.39547/) (2026)

## Abstract (arXiv)

Graphical User Interface (GUI) agents, which autonomously operate on digital interfaces through natural language instructions, hold transformative potential for accessibility, automation, and user experience. A critical aspect of their functionality is grounding - the ability to map linguistic intents to visual and structural interface elements. However, existing GUI agents often struggle to adapt to the dynamic and interconnected nature of real-world digital environments, where tasks frequently span multiple platforms and applications while also being impacted by version updates. To address this, we introduce TransBench, the first benchmark designed to systematically evaluate and enhance the transferability of GUI agents across three key dimensions: cross-version transferability (adapting to version updates), cross-platform transferability (generalizing across platforms like iOS, Android, and Web), and cross-application transferability (handling tasks spanning functionally distinct apps). TransBench includes 15 app categories with diverse functionalities, capturing essential pages across versions and platforms to enable robust evaluation. Our experiments demonstrate significant improvements in grounding accuracy, showcasing the practical utility of GUI agents in dynamic, real-world environments. Our code and data will be publicly available at GitHub.