# Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing

arXiv: https://arxiv.org/abs/2305.09434 · HTML: https://mlsanity.com/paper/2305.09434/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2023.

## Built on by (most cited first, 20 of 20)

- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents](https://mlsanity.com/paper/2407.00993/) (2024)
- [AXNav: Replaying Accessibility Tests from Natural Language](https://mlsanity.com/paper/2310.02424/) (2023)
- [Foundations and Recent Trends in Multimodal Mobile Agents: A Survey](https://mlsanity.com/paper/2411.02006/) (2024)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [UI Layout Generation with LLMs Guided by UI Grammar](https://mlsanity.com/paper/2310.15455/) (2023)
- [Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing](https://mlsanity.com/paper/2311.08649/) (2023)
- [GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent](https://mlsanity.com/paper/2412.18426/) (2024)
- [GUI Action Narrator: Where and When Did That Action Take Place?](https://mlsanity.com/paper/2406.13719/) (2024)
- [Data Analysis in the Era of Generative AI](https://mlsanity.com/paper/2409.18475/) (2024)
- [WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements](https://mlsanity.com/paper/2602.11724/) (2026)
- [ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models](https://mlsanity.com/paper/2504.02110/) (2025)
- [Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation](https://mlsanity.com/paper/2507.02306/) (2025)
- [ChatGPT and Persuasive Technologies for the Management and Delivery of Personalized Recommendations in Hotel Hospitality](https://mlsanity.com/paper/2307.14298/) (2023)
- [Towards Test Generation from Task Description for Mobile Testing with Multi-modal Reasoning](https://mlsanity.com/paper/2504.15917/) (2025)
- [GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes](https://mlsanity.com/paper/2510.04791/) (2025)
- [Model-Enhanced LLM-Driven VUI Testing of VPA Apps](https://mlsanity.com/paper/2407.02791/) (2024)
- [HxAgent: Iterative Agent Planning for End-to-End Web Application Testing](https://mlsanity.com/paper/2608.15491/) (2026)
- [RotDroid: Cross-Orientation State Equivalence Testing for Detecting GUI Rotation Bugs in Android Apps](https://mlsanity.com/paper/2608.25425/) (2026)

## Abstract (arXiv)

Mobile apps are indispensable for people's daily life, and automated GUI (Graphical User Interface) testing is widely used for app quality assurance. There is a growing interest in using learning-based techniques for automated GUI testing which aims at generating human-like actions and interactions. However, the limitations such as low testing coverage, weak generalization, and heavy reliance on training data, make an urgent need for a more effective approach to generate human-like actions to thoroughly test mobile apps. Inspired by the success of the Large Language Model (LLM), e.g., GPT-3 and ChatGPT, in natural language understanding and question answering, we formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts, and executing them to keep passing the app feedback to LLM, iterating the whole process. Within it, we extract the static context of the GUI page and the dynamic context of the iterative testing process, design prompts for inputting this information to LLM, and develop a neural matching network to decode the LLM's output into actionable steps to execute the app. We evaluate GPTDroid on 86 apps from Google Play, and its activity coverage is 71%, with 32% higher than the best baseline, and can detect 36% more bugs with faster speed than the best baseline. GPTDroid also detects 48 new bugs on the Google Play with 25 of them being confirmed/fixed. We further summarize the capabilities of GPTDroid behind the superior performance, including semantic text input, compound action, long meaningful test trace, and test case prioritization.