mlsanityData to 14 Oct 2026

UIBert: Learning Generic Multimodal Representations for UI Understanding

arXiv 2107.13731 ↗First posted 29 Jul 2021cs.CV cs.AIcited by 95

Not on the map. Why?

Cited by 33 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To improve the accessibility of smart devices and to simplify their usage, building models which understand user interfaces (UIs) and assist users to complete their tasks is critical. However, unique challenges are proposed by UI-specific characteristics, such as how to effectively leverage multimodal UI features that involve image, text, and structural metadata and how to achieve good performance when high-quality labeled data is unavailable. To address such challenges we introduce UIBert, a transformer-based joint image-text model trained through novel pre-training tasks on large-scale unlabeled UI data to learn generic feature representations for a UI and its components. Our key intuition is that the heterogeneous features in a UI are self-aligned, i.e., the image and text features of UI components, are predictive of each other. We propose five pretraining tasks utilizing this self-alignment among different features of a UI component and across various components in the same UI. We evaluate our method on nine real-world downstream UI tasks where UIBert outperforms strong multimodal baselines by up to 9.26% accuracy.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 29 Jul 2021.

Builds on

16 of 28 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  9. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  10. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  11. 2016Exploring the Limits of Language Modelingcited by 173
  12. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
4 more
  1. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  2. 2020Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learningcited by 25
  3. 2019Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learningcited by 18
  4. 2018Resolving Referring Expressions in Images With Labeled Elementscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 12; 2023: 11; 2024: 25; 2025: 33; 2026: 13.

95 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  7. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  8. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  9. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  10. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  11. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  12. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
38 more
  1. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  2. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  3. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
  4. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  5. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  6. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  7. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
  8. 2025Detect Anything via Next Point Predictioncited by 73
  9. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  10. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  11. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  12. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  13. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 48
  14. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  15. 2025Step-GUI Technical Reportcited by 39
  16. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  17. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  18. 2023Towards Flexible Multi-modal Document Modelscited by 31
  19. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  20. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  21. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  22. 2022UGIF: UI Grounded Instruction Followingcited by 25
  23. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  24. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  25. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 23
  26. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  27. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  28. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  29. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 19
  30. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18
  31. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  32. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  33. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  34. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  35. 2023A Parse-Then-Place Approach for Generating Graphic Layouts from Textual Descriptionscited by 15
  36. 2024Improving Language Understanding from Screenshotscited by 14
  37. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  38. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 13

50 most cited of 95 All 95 (JSON)