mlsanityData to 7 Oct 2026

Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

arXiv 2509.26539 ↗First posted 30 Sep 2025cs.CV cs.CL cs.LGcited by 14

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Developing autonomous agents that effectively interact with Graphic User Interfaces (GUIs) remains a challenging open problem, especially for small on-device models. In this paper, we present Ferret-UI Lite, a compact, end-to-end GUI agent that operates across diverse platforms, including mobile, web, and desktop. Utilizing techniques optimized for developing small models, we build our 3B Ferret-UI Lite agent through curating a diverse GUI data mixture from real and synthetic sources, strengthening inference-time performance through chain-of-thought reasoning and visual tool-use, and reinforcement learning with designed rewards. Ferret-UI Lite achieves competitive performance with other small-scale GUI agents. In GUI grounding, Ferret-UI Lite attains scores of 91.6%, 53.3%, and 61.2% on the ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI navigation, Ferret-UI Lite achieves success rates of 28.0% on AndroidWorld and 19.8% on OSWorld. We share our methods and lessons learned from developing compact, on-device GUI agents.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 61 of its 67 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

61 of 67 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
  2. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
  4. 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
  5. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents21 May
  6. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
  7. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
  8. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
  9. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents17 Apr
  10. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
  11. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  12. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
10 more
  1. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  4. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  6. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  8. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  10. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
27 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  3. 2024Genie: Generative Interactive Environmentscited by 706
  4. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  5. 2025Kimi-VL Technical Reportcited by 381
  6. 2025Seed1.5-VL Technical Reportcited by 367
  7. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  8. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  9. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  10. 2025Small Language Models are the Future of Agentic AIcited by 254
  11. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  12. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  13. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  14. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  15. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  16. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  17. 2025GRIT: Teaching MLLMs to Think with Imagescited by 121
  18. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  19. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  20. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  21. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  22. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  23. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  24. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  25. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  26. 2025Scaling Synthetic Task Generation for Agents via Explorationcited by 19
  27. 2025InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interactioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  2. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 30
  3. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
  4. 2026UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agentscited by 11
  5. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  6. 2026Step-level Optimization for Efficient Computer-use Agentscited by 7
  7. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 5
  8. 2026POINTS-GUI-G: GUI-Grounding Journeycited by 3
  9. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agentscited by 2
  10. 2026Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agentscited by 1
  11. 2026MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Modelscited by 1
  12. 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Groundingcited by 1
2 more
  1. 2026VUDA: Enabling Controlled Spatial Sharing of Graphics and Compute on NVIDIA GPUscited by 0
  2. 2026MintAct: A Unified Visual Agent for Digital Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TinyClick: Single-Turn Agent for Empowering GUI Automation
  2. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
  3. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning
  4. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
  5. 2026StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
  6. 2026MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration
  7. 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
  8. 2026LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning