mlsanityData to 7 Oct 2026

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

arXiv 2404.05719 ↗First posted 8 Apr 2024cs.CV cs.CL cs.HCcited by 173

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.

Key facts. Cited by 173 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 63 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

49 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understanding7 Feb
  2. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  3. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov
  4. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations7 Oct
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  6. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus29 Sep

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2023Mistral 7Bcited by 3,065
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
31 more
  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  6. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  7. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2023Language Models can Solve Computer Taskscited by 371
  10. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  11. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  12. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  13. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  14. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  15. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  16. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  17. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  18. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  19. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  20. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  21. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  22. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  23. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  24. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  25. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  26. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  27. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  28. 2018Learning to Navigate the Webcited by 60
  29. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  30. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  31. 2018Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 50; 2025: 68; 2026: 55.

173 papers.

Sort:

By area:

  1. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  2. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  4. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  6. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  7. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  8. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  9. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  10. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  11. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  12. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
38 more
  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  2. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  3. 2024Beyond Browsing: API-Based Web Agentscited by 77
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  5. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  6. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  7. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  8. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  9. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  10. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  11. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilitiescited by 57
  12. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  13. 2024Toward Visual Grounding: A Surveycited by 52
  14. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  15. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  16. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  17. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  18. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  19. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  20. 2026Code as Agent Harnesscited by 40
  21. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  22. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
  23. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  24. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  25. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  26. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  27. 2024MobileFlow: A Multimodal LLM For Mobile GUI Agentcited by 31
  28. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  29. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 31
  30. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  31. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  32. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 30
  33. 2024MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devicescited by 29
  34. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  35. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  36. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  37. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  38. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25

50 most cited of 173 All 173 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging Multimodal LLM for Inspirational User Interface Search
  2. 2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements
  3. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs
  4. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
  5. 2026Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models ↗
  6. 2024Inferring Alt-text For UI Icons With Large Language Models During App Development ↗
  7. 2026Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
  8. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning