mlsanityData to 7 Oct 2026

Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus

arXiv 2209.14927 ↗First posted 29 Sep 2022cs.CV cs.HC cs.LGcited by 62

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Mobile UI understanding is important for enabling various interaction tasks such as UI automation and accessibility. Previous mobile UI modeling often depends on the view hierarchy information of a screen, which directly provides the structural data of the UI, with the hope to bypass challenging tasks of visual modeling from screen pixels. However, view hierarchies are not always available, and are often corrupted with missing object descriptions or misaligned structure information. As a result, despite the use of view hierarchies could offer short-term gains, it may ultimately hinder the applicability and performance of the model. In this paper, we propose Spotlight, a vision-only approach for mobile UI understanding. Specifically, we enhance a vision-language model that only takes the screenshot of the UI and a region of interest on the screen -- the focus -- as the input. This general architecture of Spotlight is easily scalable and capable of performing a range of UI modeling tasks. Our experiments show that our model establishes SoTA results on several representative UI tasks and outperforms previous methods that use both screenshots and view hierarchies as inputs. Furthermore, we explore multi-task learning and few-shot prompting capacities of the proposed models, demonstrating promising results in the multi-task learning direction.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 33 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

28 of 33 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
16 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  4. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  5. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  6. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  7. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  8. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  9. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  10. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  11. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  12. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  13. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  14. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  15. 2019Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learningcited by 18
  16. 2022Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 27; 2025: 16; 2026: 7.

62 papers.

Sort:

By area:

  1. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  2. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  3. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  4. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  5. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  6. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  7. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  8. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  9. 2023Rich Human Feedback for Text-to-Image Generationcited by 139
  10. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  11. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  12. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
38 more
  1. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  2. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  3. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
  4. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  5. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  6. 2024Dual-View Visual Contextualization for Web Navigationcited by 24
  7. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  8. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18
  9. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  10. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  11. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  12. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  13. 2024Improving Language Understanding from Screenshotscited by 14
  14. 2024UIClip: A Data-driven Model for Assessing User Interface Designcited by 14
  15. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  16. 2024Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuningcited by 13
  17. 2023Automatic Macro Mining from Interaction Traces at Scalecited by 12
  18. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  19. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 12
  20. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  21. 2024GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agentcited by 11
  22. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  23. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Datacited by 10
  24. 2024Classification Done Right for Vision-Language Pre-Trainingcited by 10
  25. 2023UniAR: A Unified model for predicting human Attention and Responses on visual contentcited by 7
  26. 2026Mapping the Design Space of User Experience for Computer Use Agentscited by 7
  27. 2025CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planningcited by 6
  28. 2023UINav: A Practical Approach to Train On-Device Automation Agentscited by 5
  29. 2024Tell Me What's Next: Textual Foresight for Generic UI Representationscited by 5
  30. 2024GUing: A Mobile GUI Search Engine using a Vision-Language Modelcited by 4
  31. 2024UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activitycited by 4
  32. 2024Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfacescited by 3
  33. 2026A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agentscited by 3
  34. 2023Referring to Screen Texts with Voice Assistantscited by 2
  35. 2025Explorer: Robust Collection of Interactable GUI Elementscited by 2
  36. 2025Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypescited by 2
  37. 2025Small Models, Big Results: Achieving Superior Intent Extraction through Decompositioncited by 1
  38. 2025Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documentscited by 1

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design
  2. 2025CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design
  3. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  4. 2026GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models
  5. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
  6. 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection
  7. 2026Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models ↗
  8. 2024Training a Vision Language Model as Smartphone Assistant