Data to 14 Oct 2026

UI-TARS: Pioneering Automated GUI Interaction with Native Agents

arXiv 2501.12326 ↗First posted 21 Jan 2025cs.AI cs.CL cs.CVcited by 610

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions (e.g., keyboard and mouse operations). Unlike prevailing agent frameworks that depend on heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts and workflows, UI-TARS is an end-to-end model that outperforms these sophisticated frameworks. Experiments demonstrate its superior performance: UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating perception, grounding, and GUI task execution. Notably, in the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld, UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of GUI screenshots for context-aware understanding of UI elements and precise captioning; (2) Unified Action Modeling, which standardizes actions into a unified space across platforms and achieves precise grounding and interaction through large-scale action traces; (3) System-2 Reasoning, which incorporates deliberate reasoning into multi-step decision making, involving multiple reasoning patterns such as task decomposition, reflection thinking, milestone recognition, etc. (4) Iterative Training with Reflective Online Traces, which addresses the data bottleneck by automatically collecting, filtering, and reflectively refining new interaction traces on hundreds of virtual machines. Through iterative training and reflection tuning, UI-TARS continuously learns from its mistakes and adapts to unforeseen situations with minimal human intervention. We also analyze the evolution path of GUI agents to guide the further development of this domain.

Key facts. Cited by 610 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 68 of its 127 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

68 of 127 references.

Closest ancestors · 19

Same direction, newest first

7 more

Other references · 49

2023GPT-4 Technical Reportcited by 18,396
2024GPT-4o System Cardcited by 4,588
2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
2024OpenAI o1 System Cardcited by 1,244
2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468

37 more

2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
2024Understanding the planning of LLM agents: A surveycited by 421
2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
2024Agent Workflow Memorycited by 383
2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
2016FastText.zip: Compressing text classification modelscited by 335
2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
2024OmniParser for Pure Vision Based GUI Agentcited by 198
2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 70
2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
2023Dynamic Planning with a LLMcited by 61
2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 50
2023DroidBot-GPT: GPT-powered UI Automation for Androidcited by 47
2024Octopus v2: On-device language model for super agentcited by 47
2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
2024On the Multi-turn Instruction Following for Conversational Web Agentscited by 43
2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understandingcited by 43
2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 39
2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 38
2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
2024Octopus: On-device language model for function calling of software APIscited by 27
2024ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomycited by 23
2024Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platformscited by 13
2023WebWISE: Web Interface Control and Sequential Exploration with Large Language Modelscited by 11
2024Distributed Online Life-Long Learning (DOL3) for Multi-agent Trust and Reputation Assessment in E-commercecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 256; 2026: 354.

610 papers.

Sort:

By area:

2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 159
2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 141

38 more

2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
2025MiMo-VL Technical Reportcited by 115
2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
2025GTA1: GUI Test-time Scaling Agentcited by 106
2025Memp: Exploring Agent Procedural Memorycited by 103
2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
2025Grounded Reinforcement Learning for Visual Reasoningcited by 83
2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 76
2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
2025Paper2Poster: Towards Multimodal Poster Automation from Scientific Paperscited by 63
2025A Survey on (M)LLM-Based GUI Agentscited by 62
2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 62
2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 61
2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 61
2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 53
2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 50
2026Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoningcited by 49
2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
2026Code as Agent Harnesscited by 47
2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 46
2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 45
2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 43

50 most cited of 610 All 610 (JSON)

Similar, not cited

Close in wording, no citation either way