# Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots

arXiv: https://arxiv.org/abs/2109.08763 · HTML: https://mlsanity.com/paper/2109.08763/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2021.

## Built on by (most cited first, 30 of 32)

- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus](https://mlsanity.com/paper/2209.14927/) (2022)
- [WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs](https://mlsanity.com/paper/2404.06369/) (2024)
- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents](https://mlsanity.com/paper/2507.22827/) (2025)
- [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation](https://mlsanity.com/paper/2508.16763/) (2025)
- [Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis](https://mlsanity.com/paper/2204.02448/) (2022)
- [AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces](https://mlsanity.com/paper/2303.06782/) (2023)
- [TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents](https://mlsanity.com/paper/2502.08226/) (2025)
- [DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation](https://mlsanity.com/paper/2410.00201/) (2024)
- [UINav: A Practical Approach to Train On-Device Automation Agents](https://mlsanity.com/paper/2312.10170/) (2023)
- [Visual Prompting with Iterative Refinement for Design Critique Generation](https://mlsanity.com/paper/2412.16829/) (2024)
- [UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments](https://mlsanity.com/paper/2509.21733/) (2025)
- [UI-UG: A Unified MLLM for UI Understanding and Generation](https://mlsanity.com/paper/2509.24361/) (2025)
- [Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents](https://mlsanity.com/paper/2606.31270/) (2026)
- [MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents](https://mlsanity.com/paper/2512.12634/) (2025)
- [GreaseVision: Rewriting the Rules of the Interface](https://mlsanity.com/paper/2204.03731/) (2022)
- [ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models](https://mlsanity.com/paper/2504.02110/) (2025)
- [SpaceEditing: Integrating Human Knowledge into Deep Neural Networks via Interactive Latent Space Editing](https://mlsanity.com/paper/2212.04065/) (2022)
- [AutoGameUI: Constructing High-Fidelity GameUI via Multimodal Correspondence Matching](https://mlsanity.com/paper/2411.03709/) (2024)
- [UISearch: Graph-Based Embeddings for Multimodal Enterprise UI Screenshots Retrieval](https://mlsanity.com/paper/2511.19380/) (2025)
- [DOne: Decoupling Structure and Rendering for High-Fidelity Design-to-Code Generation](https://mlsanity.com/paper/2604.01226/) (2026)
- [Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools](https://mlsanity.com/paper/2607.22928/) (2026)

## Abstract (arXiv)

Automated understanding of user interfaces (UIs) from their pixels can improve accessibility, enable task automation, and facilitate interface design without relying on developers to comprehensively provide metadata. A first step is to infer what UI elements exist on a screen, but current approaches are limited in how they infer how those elements are semantically grouped into structured interface definitions. In this paper, we motivate the problem of screen parsing, the task of predicting UI elements and their relationships from a screenshot. We describe our implementation of screen parsing and provide an effective training procedure that optimizes its performance. In an evaluation comparing the accuracy of the generated output, we find that our implementation significantly outperforms current systems (up to 23%). Finally, we show three example applications that are facilitated by screen parsing: (i) UI similarity search, (ii) accessibility enhancement, and (iii) code generation from UI screenshots.