mlsanityData to 7 Oct 2026

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

arXiv 2101.04893 ↗First posted 13 Jan 2021cs.HCcited by 57

Not on the map. Why?

Abstract (arXiv)

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app's full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jan 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 15; 2024: 17; 2025: 10; 2026: 2.

57 papers.

Sort:

By area:

  1. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  3. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  4. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  5. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  6. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  7. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  8. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  9. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  10. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  11. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  12. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
38 more
  1. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  2. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  3. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
  4. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  5. 2023AXNav: Replaying Accessibility Tests from Natural Languagecited by 31
  6. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  7. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  8. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  9. 2023Intelligent Virtual Assistants with LLM-based Process Automationcited by 26
  10. 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedbackcited by 26
  11. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  12. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  13. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  14. 2021Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labelscited by 17
  15. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  16. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  17. 2021Understanding Mobile GUI: from Pixel-Words to Screen-Sentencescited by 13
  18. 2025WinClick: GUI Grounding with Multimodal Large Language Modelscited by 11
  19. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  20. 2022Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysiscited by 9
  21. 2023AidUI: Toward Automated Recognition of Dark Patterns in User Interfacescited by 9
  22. 2025TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agentscited by 9
  23. 2021Multimodal Icon Annotation For Mobile Applicationscited by 8
  24. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  25. 2022Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Imagescited by 5
  26. 2023UINav: A Practical Approach to Train On-Device Automation Agentscited by 5
  27. 2023An Empirical Investigation into the Use of Image Captioning for Automated Software Documentationcited by 4
  28. 2023MenuCraft: Interactive Menu System Design with Large Language Modelscited by 4
  29. 2024UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activitycited by 4
  30. 2025Agent-Initiated Interaction in Phone UI Automationcited by 4
  31. 2024The Impact of Element Ordering on LM Agent Performancecited by 3
  32. 2025AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agentcited by 3
  33. 2023Referring to Screen Texts with Voice Assistantscited by 2
  34. 2023EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learningcited by 2
  35. 2024Automated Code Fix Suggestions for Accessibility Issues in Mobile Appscited by 2
  36. 2025Explorer: Robust Collection of Interactable GUI Elementscited by 2
  37. 2022UI Layers Merger: Merging UI layers via Visual Learning and Boundary Priorcited by 1
  38. 2024PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLMcited by 1

50 most cited of 57 All 57 (JSON)