mlsanityData to 7 Oct 2026

UFO: A UI-Focused Agent for Windows OS Interaction

arXiv 2402.07939 ↗First posted 8 Feb 2024cs.HC cs.AI cs.CLcited by 147

Not on the map. Why?

Cited by 95 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce UFO, an innovative UI-Focused agent to fulfill user requests tailored to applications on Windows OS, harnessing the capabilities of GPT-Vision. UFO employs a dual-agent framework to meticulously observe and analyze the graphical user interface (GUI) and control information of Windows applications. This enables the agent to seamlessly navigate and operate within individual applications and across them to fulfill user requests, even when spanning multiple applications. The framework incorporates a control interaction module, facilitating action grounding without human intervention and enabling fully automated execution. Consequently, UFO transforms arduous and time-consuming processes into simple tasks achievable solely through natural language commands. We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios reflective of users' daily usage. The results, derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFO in fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS environment. The open-source code for UFO is available on https://github.com/microsoft/UFO.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

32 of 47 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Segment Anythingcited by 8,158
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
20 more
  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  6. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  7. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  8. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  9. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  10. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  11. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  12. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  13. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  14. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  15. 2023TaskWeaver: A Code-First Agent Frameworkcited by 85
  16. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  17. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  18. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  19. 2023Xpert: Empowering Incident Management with Query Recommendations via Large Language Modelscited by 22
  20. 2023Assess and Summarize: Improve Outage Understanding with Large Language Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 46; 2025: 70; 2026: 31.

147 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  4. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  6. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  7. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  8. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  9. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  10. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  11. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  12. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
38 more
  1. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  2. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  4. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  5. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  6. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  7. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  8. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  9. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  10. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  11. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  12. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  13. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  14. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  15. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  16. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  17. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  18. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  19. 2024TinyAgent: Function Calling at the Edgecited by 55
  20. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 55
  21. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  22. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 51
  23. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  24. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  25. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 44
  26. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  27. 2025PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PCcited by 39
  28. 2025LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automationcited by 39
  29. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 38
  30. 2025UFO2: The Desktop AgentOScited by 38
  31. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
  32. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  33. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  34. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  35. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  36. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  37. 2024MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devicescited by 29
  38. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29

50 most cited of 147 All 147 (JSON)