mlsanityData to 7 Oct 2026

Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots

arXiv 2109.08763 ↗First posted 17 Sep 2021cs.HCcited by 32

Not on the map. Why?

Abstract (arXiv)

Automated understanding of user interfaces (UIs) from their pixels can improve accessibility, enable task automation, and facilitate interface design without relying on developers to comprehensively provide metadata. A first step is to infer what UI elements exist on a screen, but current approaches are limited in how they infer how those elements are semantically grouped into structured interface definitions. In this paper, we motivate the problem of screen parsing, the task of predicting UI elements and their relationships from a screenshot. We describe our implementation of screen parsing and provide an effective training procedure that optimizes its performance. In an evaluation comparing the accuracy of the generated output, we find that our implementation significantly outperforms current systems (up to 23%). Finally, we show three example applications that are facilitated by screen parsing: (i) UI similarity search, (ii) accessibility enhancement, and (iii) code generation from UI screenshots.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Sep 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 4; 2024: 7; 2025: 9; 2026: 5.

32 papers.

Sort:

By area:

  1. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  2. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  3. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  4. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  5. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  6. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  7. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 43
  8. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  9. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  10. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
  11. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agentscited by 23
  12. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
20 more
  1. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  2. 2022Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysiscited by 9
  3. 2023AidUI: Toward Automated Recognition of Dark Patterns in User Interfacescited by 9
  4. 2025TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agentscited by 9
  5. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  6. 2023UINav: A Practical Approach to Train On-Device Automation Agentscited by 5
  7. 2024Visual Prompting with Iterative Refinement for Design Critique Generationcited by 5
  8. 2025UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environmentscited by 5
  9. 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agentscited by 5
  10. 2025UI-UG: A Unified MLLM for UI Understanding and Generationcited by 4
  11. 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agentscited by 3
  12. 2022GreaseVision: Rewriting the Rules of the Interfacecited by 2
  13. 2025ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Modelscited by 2
  14. 2022SpaceEditing: Integrating Human Knowledge into Deep Neural Networks via Interactive Latent Space Editingcited by 1
  15. 2024AutoGameUI: Constructing High-Fidelity GameUI via Multimodal Correspondence Matchingcited by 1
  16. 2025UISearch: Graph-Based Embeddings for Multimodal Enterprise UI Screenshots Retrievalcited by 0
  17. 2026DOne: Decoupling Structure and Rendering for High-Fidelity Design-to-Code Generationcited by 0
  18. 2026Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Toolscited by 0
  19. 2026Enabling Creative Exploration for Vibe Design Agentscited by 0
  20. 2026A Benchmarking Framework for Context-aware XR Interfacescited by 0