mlsanityData to 14 Oct 2026

Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

arXiv 2410.18967 ↗First posted 24 Oct 2024cs.CV cs.CL cs.LGcited by 51

Not on the map. Why?

Cited by 29 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks × 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 51 of its 62 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

51 of 62 references.

References · 51

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  10. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
38 more
  1. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  2. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  3. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  4. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  6. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  7. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  8. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
  9. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  10. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  12. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
  13. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  14. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  15. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  16. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  17. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  18. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  19. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 118
  20. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  21. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  22. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
  23. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  24. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  25. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  26. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  27. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  28. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 68
  29. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  30. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 66
  31. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agentscited by 63
  32. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  33. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 47
  34. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  35. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  36. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understandingcited by 43
  37. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  38. 2024MobileFlow: A Multimodal LLM For Mobile GUI Agentcited by 32

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 26; 2026: 16.

51 papers.

Sort:

By area:

  1. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  3. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  4. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  5. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
  6. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  7. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  8. 2024Toward Visual Grounding: A Surveycited by 52
  9. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  10. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  11. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  12. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
38 more
  1. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 31
  2. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  3. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  4. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  5. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  6. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 19
  7. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  8. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  9. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10
  10. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operationcited by 9
  11. 2024OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inferencecited by 8
  12. 2025Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questionscited by 8
  13. 2024Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refiningcited by 7
  14. 2026Mapping the Design Space of User Experience for Computer Use Agentscited by 7
  15. 2025OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?cited by 6
  16. 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Taskscited by 6
  17. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 6
  18. 2025STEVE: A Step Verification Pipeline for Computer-use Agent Trainingcited by 5
  19. 2025FineState-Bench: A Comprehensive Benchmark for Fine-Grained State Control in GUI Agentscited by 5
  20. 2025CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMscited by 5
  21. 2026FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Settingcited by 5
  22. 2025DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agentscited by 4
  23. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  24. 2025FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Datacited by 3
  25. 2025PixelWeb: The First Web GUI Dataset with Pixel-Wise Labelscited by 3
  26. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  27. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agentscited by 3
  28. 2025MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Modelscited by 2
  29. 2026Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decompositioncited by 2
  30. 2026IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agentscited by 2
  31. 2024Automatically Detecting Online Deceptive Patternscited by 1
  32. 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
  33. 2026FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolationcited by 1
  34. 2026Software Engineering for and with GUI Agentcited by 1
  35. 2025Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approachcited by 0
  36. 2026TVWorld: Foundations for Remote-Control TV Agentscited by 0
  37. 2026Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshotscited by 0
  38. 2026FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolationcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025UI-UG: A Unified MLLM for UI Understanding and Generation
  2. 2025Structuring GUI Elements through Vision Language Models: Towards Action Space Generation
  3. 2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements
  4. 2026Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
  5. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
  6. 2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
  7. 2025GUIrilla: A Scalable Framework for Automated Desktop UI Exploration
  8. 2025Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding