# Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions

arXiv: https://arxiv.org/abs/2310.15780 · HTML: https://mlsanity.com/paper/2310.15780/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2023.

## Built on by (most cited first, 30 of 41)

- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [GUI Agents: A Survey](https://mlsanity.com/paper/2412.13501/) (2024)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks](https://mlsanity.com/paper/2508.13143/) (2025)
- [Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue](https://mlsanity.com/paper/2409.08330/) (2024)
- [NaviQAte: Functionality-Guided Web Application Navigation](https://mlsanity.com/paper/2409.10741/) (2024)
- [From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis](https://mlsanity.com/paper/2502.11919/) (2025)
- [LLM-Explorer: Towards Efficient and Affordable LLM-based Exploration for Mobile Apps](https://mlsanity.com/paper/2505.10593/) (2025)
- [LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities](https://mlsanity.com/paper/2601.09822/) (2026)
- [Retrieval-Augmented Test Generation: How Far Are We?](https://mlsanity.com/paper/2409.12682/) (2024)
- [On the Effectiveness of Large Language Models in Writing Alloy Formulas](https://mlsanity.com/paper/2502.15441/) (2025)
- [Is LLMs Hallucination Usable? LLM-based Negative Reasoning for Fake News Detection](https://mlsanity.com/paper/2503.09153/) (2025)
- [Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Development](https://mlsanity.com/paper/2509.25297/) (2025)
- [Are Autonomous Web Agents Good Testers?](https://mlsanity.com/paper/2504.01495/) (2025)
- [AI Agents for Web Testing: A Case Study in the Wild](https://mlsanity.com/paper/2509.05197/) (2025)
- [Agent for User: Testing Multi-User Interactive Features in TikTok](https://mlsanity.com/paper/2504.15474/) (2025)
- [CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models](https://mlsanity.com/paper/2512.07890/) (2025)
- [WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements](https://mlsanity.com/paper/2602.11724/) (2026)
- [WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing](https://mlsanity.com/paper/2603.25226/) (2026)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [OBsmith: LLM-Powered JavaScript Obfuscator Testing](https://mlsanity.com/paper/2510.10066/) (2025)
- [WIP: Assessing the Effectiveness of ChatGPT in Preparatory Testing Activities](https://mlsanity.com/paper/2503.03951/) (2025)
- [Breaking Single-Tester Limits: Multi-Agent LLMs for Multi-User Feature Testing](https://mlsanity.com/paper/2506.17539/) (2025)
- [Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough](https://mlsanity.com/paper/2512.03568/) (2025)
- [SolAgent: A Specialized Multi-Agent Framework for Solidity Code Generation](https://mlsanity.com/paper/2601.23009/) (2026)
- [Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems](https://mlsanity.com/paper/2604.12231/) (2026)
- [DeepDiscovery: A Location-Inference Framework for Task-Level Repository Understanding](https://mlsanity.com/paper/2606.22906/) (2026)
- [From Exploration to Revelation: Detecting Dark Patterns in Mobile Apps](https://mlsanity.com/paper/2411.18084/) (2024)

## Abstract (arXiv)

Automated Graphical User Interface (GUI) testing plays a crucial role in ensuring app quality, especially as mobile applications have become an integral part of our daily lives. Despite the growing popularity of learning-based techniques in automated GUI testing due to their ability to generate human-like interactions, they still suffer from several limitations, such as low testing coverage, inadequate generalization capabilities, and heavy reliance on training data. Inspired by the success of Large Language Models (LLMs) like ChatGPT in natural language understanding and question answering, we formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts, and executing them to keep passing the app feedback to LLM, iterating the whole process. Within this framework, we have also introduced a functionality-aware memory prompting mechanism that equips the LLM with the ability to retain testing knowledge of the whole process and conduct long-term, functionality-based reasoning to guide exploration. We evaluate it on 93 apps from Google Play and demonstrate that it outperforms the best baseline by 32% in activity coverage, and detects 31% more bugs at a faster rate. Moreover, GPTDroid identify 53 new bugs on Google Play, of which 35 have been confirmed and fixed.