# Large Language Models for Mobile GUI Text Input Generation: An Empirical Study

arXiv: https://arxiv.org/abs/2404.08948 · HTML: https://mlsanity.com/paper/2404.08948/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2024.

## Built on by (most cited first, 3 of 3)

- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning](https://mlsanity.com/paper/2406.14056/) (2024)
- [Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach](https://mlsanity.com/paper/2606.13192/) (2026)

## Abstract (arXiv)

Mobile apps have become essential, making quality assurance increasingly important. GUI testing is widely used for automated exploration, yet text-input components remain a major obstacle, as many UI pages require semantically appropriate text inputs before proceeding. Large Language Models have shown promise in generating context-aware text, but the effectiveness of different UI representations, feedback mechanisms, and human intervention remains unclear. This paper presents a large-scale empirical study addressing these gaps. We evaluate nine state-of-the-art LLMs across 115 real-world apps, comparing three UI-context prompting settings: extracted textual context, UI-hierarchy XML, and screenshot-based vision input. Results show extracted context and XML achieve comparable page-pass-through rates of 71.4% and 71.0%, while vision-based input reaches 65.1% but incurs substantially higher token costs. In bug-detection experiments with 37 real-world text-input bugs, LLMs generating invalid inputs detect about 51% of issues across all evaluated models. A feedback-enhanced protocol, incorporating execution outcomes into subsequent attempts, improves average PPTRs to 69.2-73.8% and raises bug-detection rates to 51.0-64.5%. Human testers further refine inputs, yielding additional gains. We integrate the process into DroidBot, augmenting its UI-exploration capabilities. We derive actionable insights on context selection, cost-effectiveness, feedback strategies, and human-LLM collaboration, advancing both knowledge and practice in Android testing.