mlsanityData to 7 Oct 2026

VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling

arXiv 2112.05692 ↗First posted 10 Dec 2021cs.CV cs.AI cs.HCcited by 26

Not on the map. Why?

Abstract (arXiv)

User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language. The tasks are also diverse, including object detection, language generation and grounding. In this paper, we present VUT, a Versatile UI Transformer that takes multimodal input and simultaneously accomplishes 5 distinct tasks with the same model. Our model consists of a multimodal Transformer encoder that jointly encodes UI images and structures, and performs UI object detection when the UI structures are absent in the input. Our model also consists of an auto-regressive Transformer model that encodes the language input and decodes output, for both question-answering and command grounding with respect to the UI. Our experiments show that for most of the tasks, when trained jointly for multi-tasks, VUT substantially reduces the number of models and footprints needed for performing multiple tasks, while achieving accuracy exceeding or on par with baseline models trained for each individual task.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

23 of 28 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2015Pointer Networkscited by 882
  10. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  11. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  12. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
11 more
  1. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  2. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  3. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  4. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  5. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  6. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  7. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  8. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  9. 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
  10. 2018Mapping Natural Language Commands to Web Elementscited by 23
  11. 2019Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 2; 2024: 10; 2025: 5; 2026: 2.

26 papers.

Sort:

By area:

  1. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  2. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  4. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  5. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  6. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  7. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  8. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  9. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  10. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  11. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  12. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
14 more
  1. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  2. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  3. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  4. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  5. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  6. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  7. 2022MUG: Interactive Multimodal Grounding on User Interfacescited by 9
  8. 2025KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generationcited by 9
  9. 2022Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Unified Vision-and-Language BERTscited by 5
  10. 2023Referring to Screen Texts with Voice Assistantscited by 2
  11. 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
  12. 2024Computer User Interface Understanding. A New Dataset and a Learning Frameworkcited by 0
  13. 2025GAIR: GUI Automation via Information-Joint Reasoning and Group Reflectioncited by 0
  14. 2026Multi-modal user interface control detection using cross-attentioncited by 0