# DroidBot-GPT: GPT-powered UI Automation for Android

arXiv: https://arxiv.org/abs/2304.07061 · HTML: https://mlsanity.com/paper/2304.07061/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 14 Apr 2023.

## Builds on (most cited first, 8 of 8)

- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [Mapping Natural Language Commands to Web Elements](https://mlsanity.com/paper/1808.09132/) (2018)
- [MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels](https://mlsanity.com/paper/2111.15031/) (2021)
- [Lexi: Self-Supervised Learning of the UI Language](https://mlsanity.com/paper/2301.10165/) (2023)
- [Screen Correspondence: Mapping Interchangeable Elements between UIs](https://mlsanity.com/paper/2301.08372/) (2023)

## Built on by (most cited first, 30 of 47)

- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [Large Multimodal Agents: A Survey](https://mlsanity.com/paper/2402.15116/) (2024)
- [AppAgent v2: Advanced Agent for Flexible Mobile Interactions](https://mlsanity.com/paper/2408.11824/) (2024)
- [Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation](https://mlsanity.com/paper/2312.03003/) (2023)
- [Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents](https://mlsanity.com/paper/2407.00993/) (2024)
- [ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation](https://mlsanity.com/paper/2312.13108/) (2023)
- [AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation](https://mlsanity.com/paper/2412.18116/) (2024)
- [A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions](https://mlsanity.com/paper/2501.16150/) (2025)
- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [Foundations and Recent Trends in Multimodal Mobile Agents: A Survey](https://mlsanity.com/paper/2411.02006/) (2024)
- [Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation](https://mlsanity.com/paper/2506.04614/) (2025)
- [MobileWorldBench: Towards Semantic World Modeling For Mobile Agents](https://mlsanity.com/paper/2512.14014/) (2025)
- [MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions](https://mlsanity.com/paper/2502.16796/) (2025)
- [Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing](https://mlsanity.com/paper/2311.08649/) (2023)
- [SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents](https://mlsanity.com/paper/2507.00841/) (2025)
- [Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents](https://mlsanity.com/paper/2505.14418/) (2025)
- [GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?](https://mlsanity.com/paper/2510.20333/) (2025)
- [E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion](https://mlsanity.com/paper/2406.14250/) (2024)
- [MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science](https://mlsanity.com/paper/2501.10768/) (2025)
- [From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents](https://mlsanity.com/paper/2505.12981/) (2025)
- [Identifying User Goals from UI Trajectories](https://mlsanity.com/paper/2406.14314/) (2024)
- [EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automation](https://mlsanity.com/paper/2505.05440/) (2025)
- [MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions](https://mlsanity.com/paper/2508.09057/) (2025)
- [GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph](https://mlsanity.com/paper/2601.17418/) (2026)
- [Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation](https://mlsanity.com/paper/2511.12254/) (2025)
- [LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark](https://mlsanity.com/paper/2504.13805/) (2025)

## Abstract (arXiv)

This paper introduces DroidBot-GPT, a tool that utilizes GPT-like large language models (LLMs) to automate the interactions with Android mobile applications. Given a natural language description of a desired task, DroidBot-GPT can automatically generate and execute actions that navigate the app to complete the task. It works by translating the app GUI state information and the available actions on the smartphone screen to natural language prompts and asking the LLM to make a choice of actions. Since the LLM is typically trained on a large amount of data including the how-to manuals of diverse software applications, it has the ability to make reasonable choices of actions based on the provided information. We evaluate DroidBot-GPT with a self-created dataset that contains 33 tasks collected from 17 Android applications spanning 10 categories. It can successfully complete 39.39% of the tasks, and the average partial completion progress is about 66.76%. Given the fact that our method is fully unsupervised (no modification required from both the app and the LLM), we believe there is great potential to enhance automation performance with better app development paradigms and/or custom model training.