mlsanityData to 14 Oct 2026

Lexi: Self-Supervised Learning of the UI Language

arXiv 2301.10165 ↗First posted 23 Jan 2023cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Humans can learn to operate the user interface (UI) of an application by reading an instruction manual or how-to guide. Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text. We explore how to leverage this data to learn generic visio-linguistic representations of UI screens and their components. These representations are useful in many real applications, such as accessibility, voice navigation, and task automation. Prior UI representation models rely on UI metadata (UI trees and accessibility labels), which is often missing, incompletely defined, or not accessible. We avoid such a dependency, and propose Lexi, a pre-trained vision and language model designed to handle the unique features of UI screens, including their text richness and context sensitivity. To train Lexi we curate the UICaption dataset consisting of 114k UI images paired with descriptions of their functionality. We evaluate Lexi on four tasks: UI action entailment, instruction-based UI image retrieval, grounding referring expressions, and UI entity recognition.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 23 Jan 2023.

Builds on

31 of 54 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
19 more
  1. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  2. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  3. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  4. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  5. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  6. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 151
  7. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  8. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
  9. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  10. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  11. 2018Learning to Navigate the Webcited by 60
  12. 2016Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representationscited by 55
  13. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  14. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  15. 2018Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Appscited by 36
  16. 2020Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?cited by 29
  17. 2020Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learningcited by 25
  18. 2018Mapping Natural Language Commands to Web Elementscited by 23
  19. 2020FLIN: A Flexible Natural Language Interface for Web Navigationcited by 18

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 5; 2025: 1.

10 papers.

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  4. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  5. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  6. 2023DroidBot-GPT: GPT-powered UI Automation for Androidcited by 47
  7. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  8. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  9. 2023UINav: A Practical Approach to Train On-Device Automation Agentscited by 5
  10. 2024Instructional Text Across Disciplines: A Survey of Representations, Downstream Tasks, and Open Challenges Toward Capable AI Agentscited by 1