mlsanityData to 7 Oct 2026

DroidBot-GPT: GPT-powered UI Automation for Android

arXiv 2304.07061 ↗First posted 14 Apr 2023cs.SE cs.AIcited by 47

Not on the map. Why?

Cited by 30 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces DroidBot-GPT, a tool that utilizes GPT-like large language models (LLMs) to automate the interactions with Android mobile applications. Given a natural language description of a desired task, DroidBot-GPT can automatically generate and execute actions that navigate the app to complete the task. It works by translating the app GUI state information and the available actions on the smartphone screen to natural language prompts and asking the LLM to make a choice of actions. Since the LLM is typically trained on a large amount of data including the how-to manuals of diverse software applications, it has the ability to make reasonable choices of actions based on the provided information. We evaluate DroidBot-GPT with a self-created dataset that contains 33 tasks collected from 17 Android applications spanning 10 categories. It can successfully complete 39.39% of the tasks, and the average partial completion progress is about 66.76%. Given the fact that our method is fully unsupervised (no modification required from both the app and the LLM), we believe there is great potential to enhance automation performance with better app development paradigms and/or custom model training.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 8 of its 23 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

8 of 23 references.

References · 8

  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  4. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  5. 2018Mapping Natural Language Commands to Web Elementscited by 23
  6. 2021MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labelscited by 17
  7. 2023Lexi: Self-Supervised Learning of the UI Languagecited by 10
  8. 2023Screen Correspondence: Mapping Interchangeable Elements between UIscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 12; 2025: 20; 2026: 11.

47 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  4. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  5. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  6. 2024Large Multimodal Agents: A Surveycited by 99
  7. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  8. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
  9. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  10. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  11. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  12. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
35 more
  1. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 31
  2. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  3. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  4. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  5. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  6. 2023Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testingcited by 15
  7. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 14
  8. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
  9. 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
  10. 2024E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTioncited by 11
  11. 2025MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Sciencecited by 10
  12. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  13. 2024Identifying User Goals from UI Trajectoriescited by 9
  14. 2025MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructionscited by 7
  15. 2025EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automationcited by 6
  16. 2025Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automationcited by 5
  17. 2026GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graphcited by 5
  18. 2026Learning with Challenges: Adaptive Difficulty-Aware Data Generation for Mobile GUI Agent Trainingcited by 4
  19. 2026SecAgent: Efficient Mobile GUI Agent with Semantic Contextcited by 4
  20. 2026What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoningcited by 4
  21. 2024Human-Centered LLM-Agent User Interface: A Position Papercited by 3
  22. 2025LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmarkcited by 3
  23. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  24. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  25. 2024GUI Test Migration via Abstraction and Concretizationcited by 1
  26. 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
  27. 2026Software Engineering for and with GUI Agentcited by 1
  28. 2025Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedbackcited by 0
  29. 2025Cybernaut: Towards Reliable Web Automationcited by 0
  30. 2025Measuring the Security of Mobile LLM Agents under Adversarial Prompts from Untrusted Third-Party Channelscited by 0
  31. 2026DynamicsLLM: a Dynamic Analysis-based Tool for Generating Intelligent Execution Traces Using LLMs to Detect Android Behavioural Code Smellscited by 0
  32. 2026Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshotscited by 0
  33. 2026A Comprehensive Study of Implementation Bugs in Multi-modal Agentscited by 0
  34. 2026DragonCrawl: A Generative, Intent-Based Framework for Scalable Mobile End-to-End Testingcited by 0
  35. 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agentscited by 0