mlsanityData to 7 Oct 2026

ScreenAI: A Vision-Language Model for UI and Infographics Understanding

arXiv 2402.04615 ↗First posted 7 Feb 2024cs.CV cs.AIcited by 97

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Screen user interfaces (UIs) and infographics, sharing similar visual language and design principles, play important roles in human communication and human-machine interaction. We introduce ScreenAI, a vision-language model that specializes in UI and infographics understanding. Our model improves upon the PaLI architecture with the flexible patching strategy of pix2struct and is trained on a unique mixture of datasets. At the heart of this mixture is a novel screen annotation task in which the model has to identify the type and location of UI elements. We use these text annotations to describe screens to Large Language Models and automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale. We run ablation studies to demonstrate the impact of these design choices. At only 5B parameters, ScreenAI achieves new state-of-the-artresults on UI- and infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and InfographicVQA) compared to models of similar size. Finally, we release three new datasets: one focused on the screen annotation task and two others focused on question answering.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 68 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

51 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus29 Sep

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
38 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  6. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  7. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  8. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  9. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  10. 2022UL2: Unifying Language Learning Paradigmscited by 292
  11. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  12. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  13. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  14. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  15. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  16. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  17. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  18. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  19. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  20. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  21. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  22. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  23. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  24. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  25. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
  26. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  27. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  28. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  29. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  30. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  31. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  32. 2022PreSTU: Pre-Training for Scene-Text Understandingcited by 27
  33. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  34. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  35. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  36. 2023MaMMUT: A Simple Architecture for Joint Learning for MultiModal Taskscited by 24
  37. 2021Multimodal Icon Annotation For Mobile Applicationscited by 8
  38. 2018Resolving Referring Expressions in Images With Labeled Elementscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 40; 2026: 30.

97 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  3. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  4. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  5. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  6. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  7. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  8. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  9. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  10. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  11. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  12. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
38 more
  1. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  2. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  3. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  4. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  5. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  6. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  7. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  8. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  9. 2024MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devicescited by 29
  10. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  11. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  12. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  13. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  14. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  15. 2024Large Action Models: From Inception to Implementationcited by 18
  16. 2025Redundancy Principles for MLLMs Benchmarkscited by 18
  17. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 18
  18. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  19. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 17
  20. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 17
  21. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  22. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  23. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  24. 2025ReachAgent: Enhancing Mobile Agent via Page Reaching and Operationcited by 13
  25. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigationcited by 13
  26. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  27. 2024WebQuest: A Benchmark for Multimodal QA on Web Page Sequencescited by 12
  28. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 12
  29. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  30. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  31. 2024WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Taskscited by 11
  32. 2024ClickAgent: Enhancing UI Location Capabilities of Autonomous Agentscited by 11
  33. 2025Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Taskscited by 11
  34. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11
  35. 2024WEPO: Web Element Preference Optimization for LLM-based Web Navigationcited by 10
  36. 2025AndroidGen: Building an Android Language Agent under Data Scarcitycited by 10
  37. 2024VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understandingcited by 8
  38. 2025TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environmentscited by 8

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agent
  2. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
  3. 2023An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation
  4. 2026Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models ↗
  5. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Charts
  6. 2025CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design
  7. 2024Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
  8. 2023CogAgent: A Visual Language Model for GUI Agents