# Never-ending Learning of User Interfaces

arXiv: https://arxiv.org/abs/2308.08726 · HTML: https://mlsanity.com/paper/2308.08726/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2023.

## Built on by (most cited first, 9 of 9)

- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation](https://mlsanity.com/paper/2404.16054/) (2024)
- [UIClip: A Data-driven Model for Assessing User Interface Design](https://mlsanity.com/paper/2404.12500/) (2024)
- [DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation](https://mlsanity.com/paper/2410.00201/) (2024)
- [GUing: A Mobile GUI Search Engine using a Vision-Language Model](https://mlsanity.com/paper/2405.00145/) (2024)
- [UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity](https://mlsanity.com/paper/2409.04081/) (2024)
- [Automatically Detecting Online Deceptive Patterns](https://mlsanity.com/paper/2411.07441/) (2024)
- [GhostUI: Unveiling Hidden Interactions in Mobile UI](https://mlsanity.com/paper/2601.19258/) (2026)
- [Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools](https://mlsanity.com/paper/2607.22928/) (2026)

## Abstract (arXiv)

Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human crowd-workers, a process that is costly and surprisingly error-prone for certain tasks. For example, it is possible to guess if a UI element is "tappable" from a screenshot (i.e., based on visual signifiers) or from potentially unreliable metadata (e.g., a view hierarchy), but one way to know for certain is to programmatically tap the UI element and observe the effects. We built the Never-ending UI Learner, an app crawler that automatically installs real apps from a mobile app store and crawls them to discover new and challenging training examples to learn from. The Never-ending UI Learner has crawled for more than 5,000 device-hours, performing over half a million actions on 6,000 apps to train three computer vision models for i) tappability prediction, ii) draggability prediction, and iii) screen similarity.