Can LLMs Test Terminal User Interfaces?
Not on the map. Why?
Abstract (arXiv)
Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 23 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
10 of 23 references.
References · 10
- 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
- 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103
- 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 76
- 2023Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisionscited by 41
- 2025Guidelines for Empirical Studies in Software Engineering involving Large Language Modelscited by 21
- 2015Automated Test Input Generation for Android: Are We There Yet?cited by 10
- 2021Deep Reinforcement Learning for Black-Box Testing of Android Appscited by 10
- 2021An Empirical Study of Automated Unit Test Generation for Pythoncited by 10
- 2023Testing the Limits: Unusual Text Inputs Generation for Mobile App Crash Detection with Large Language Modelcited by 10
- 2021FLACOCO: Fault Localization for Java based on Industry-grade Coveragecited by 3
Built on by
Not cited yet.