mlsanityData to 7 Oct 2026

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

arXiv 2310.04869 ↗First posted 7 Oct 2023cs.HC cs.AI cs.CLcited by 17

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations, and it can be applied to any dataset of UI screenshots. We generate a dataset of 335K conversational examples paired with UIs that cover Q&A, UI descriptions, and planning, and use it to fine-tune a conversational VLM for UI tasks. To assess the performance of our model, we benchmark it on UI element detection tasks, evaluate response quality, and showcase its applicability to multi-step UI navigation and planning.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 82 references that are in the map. First posted on arXiv 7 Oct 2023.

Builds on

26 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus29 Sep

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2023Instruction Tuning with GPT-4cited by 676
  11. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 113
  12. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
13 more
  1. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  2. 2021VINS: Visual Search for Mobile User Interface Designcited by 37
  3. 2018Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Appscited by 36
  4. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  5. 2020Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?cited by 29
  6. 2023Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testingcited by 20
  7. 2019Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learningcited by 18
  8. 2021Wireframe-Based UI Design Search Through Image Autoencodercited by 17
  9. 2022Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysiscited by 9
  10. 2019ORC Layout: Adaptive GUI Layout with OR-Constraintscited by 6
  11. 2020ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraintscited by 6
  12. 2022ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraintscited by 6
  13. 2021Rediscovering Affordance: A Reinforcement Learning Perspectivecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 7; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  3. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  4. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  5. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  6. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  7. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  8. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  9. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  10. 2026Mapping the Design Space of User Experience for Computer Use Agentscited by 7
  11. 2024Graph4GUI: Graph Neural Networks for Representing Graphical User Interfacescited by 6
  12. 2025Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypescited by 2
5 more
  1. 2026IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agentscited by 2
  2. 2024GUI Test Migration via Abstraction and Concretizationcited by 1
  3. 2025Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthroughcited by 1
  4. 2026Bridging Gulfs in UI Generation through Semantic Guidancecited by 1
  5. 2025Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Structuring GUI Elements through Vision Language Models: Towards Action Space Generation
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2024The Revolution of Multimodal Large Language Models: A Survey
  6. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understanding
  7. 2024Building and better understanding vision-language models: insights and future directions
  8. 2025UI-UG: A Unified MLLM for UI Understanding and Generation