mlsanityData to 7 Oct 2026

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

arXiv 2108.03353 ↗First posted 7 Aug 2021cs.HC cs.AI cs.LGcited by 119

Not on the map. Why?

Cited by 37 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be useful for many language-based application scenarios. We present Screen2Words, a novel screen summarization approach that automatically encapsulates essential information of a UI screen into a coherent language phrase. Summarizing mobile screens requires a holistic understanding of the multi-modal data of mobile UIs, including text, image, structures as well as UI semantics, motivating our multi-modal learning approach. We collected and analyzed a large-scale screen summarization dataset annotated by human workers. Our dataset contains more than 112k language summarization across ~22k unique UI screens. We then experimented with a set of deep models with different configurations. Our evaluation of these models with both automatic accuracy metrics and human rating shows that our approach can generate high-quality summaries for mobile screens. We demonstrate potential use cases of Screen2Words and open-source our dataset and model to lay the foundations for further bridging language and user interfaces.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 7 Aug 2021.

Builds on

26 of 53 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2018Object Hallucination in Image Captioningcited by 661
  11. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  12. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
14 more
  1. 2016Visual Storytellingcited by 176
  2. 2015Sequence to Sequence -- Video to Textcited by 157
  3. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 130
  4. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  5. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  6. 2019Neural Design Network: Graphic Layout Generation with Constraintscited by 83
  7. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  8. 2018No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytellingcited by 46
  9. 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
  10. 2021VINS: Visual Search for Mobile User Interface Designcited by 37
  11. 2020Predicting Visual Importance Across Graphic Design Typescited by 30
  12. 2019Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrationscited by 26
  13. 2019Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learningcited by 18
  14. 2019Open Challenges of Blind People using Smartphonescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 13; 2024: 36; 2025: 39; 2026: 21.

119 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  4. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  6. 2024What matters when building vision-language models?cited by 321
  7. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  8. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  9. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  10. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  11. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  12. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
38 more
  1. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  2. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  3. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  4. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  5. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  6. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  7. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  8. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  9. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  10. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  11. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
  12. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  13. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  14. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  15. 2025FineVision: Open Data Is All You Needcited by 51
  16. 2022Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summariescited by 47
  17. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  18. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  19. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  20. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  21. 2022PreSTU: Pre-Training for Scene-Text Understandingcited by 27
  22. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructionscited by 27
  23. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  24. 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedbackcited by 26
  25. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  26. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  27. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  28. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  29. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  30. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  31. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 18
  32. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  33. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  34. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  35. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  36. 2023UI Layout Generation with LLMs Guided by UI Grammarcited by 15
  37. 2024UIClip: A Data-driven Model for Assessing User Interface Designcited by 14
  38. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14

50 most cited of 119 All 119 (JSON)