mlsanityData to 14 Oct 2026

Aria-UI: Visual Grounding for GUI Instructions

arXiv 2412.16256 ↗First posted 20 Dec 2024cs.HC cs.AIcited by 140

Not on the map. Why?

Cited by 97 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant challenge due to reliance on HTML or AXTree inputs. In this paper, we introduce Aria-UI, a large multimodal model specifically designed for GUI grounding. Aria-UI adopts a pure-vision approach, eschewing reliance on auxiliary inputs. To adapt to heterogeneous planning instructions, we propose a scalable data pipeline that synthesizes diverse and high-quality instruction samples for grounding. To handle dynamic contexts in task performing, Aria-UI incorporates textual and text-image interleaved action histories, enabling robust context-aware reasoning for grounding. Aria-UI sets new state-of-the-art results across offline and online agent benchmarks, outperforming both vision-only and AXTree-reliant baselines. We release all training data and model checkpoints to foster further research at https://ariaui.github.io.

Key facts. Cited by 140 papers in the map (data through 14 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

47 of 63 references.

References · 47

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
35 more
  1. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  2. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  3. 2023Segment Everything Everywhere All at Oncecited by 548
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  5. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  6. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  7. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  8. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  9. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  10. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  11. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  12. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  13. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  14. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  15. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  16. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  17. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  18. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  19. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  20. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  21. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  22. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  23. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  24. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  25. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  26. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  27. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  28. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  29. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
  30. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  31. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  32. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  33. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  34. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  35. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 88; 2026: 52.

140 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  2. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
  3. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 159
  4. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  5. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  6. 2025GTA1: GUI Test-time Scaling Agentcited by 106
  7. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  8. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  9. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  10. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 81
  11. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  12. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
38 more
  1. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
  2. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  3. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  4. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  5. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 50
  6. 2026Code as Agent Harnesscited by 47
  7. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  8. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  9. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 42
  10. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  11. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  12. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 35
  13. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 33
  14. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  15. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
  16. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 31
  17. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  18. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 23
  19. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  20. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
  21. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  22. 2025RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Userscited by 21
  23. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 19
  24. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  25. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
  26. 2025ScaleTrack: Scaling and back-tracking Automated GUI Agentscited by 16
  27. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  28. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  29. 2025Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinementcited by 15
  30. 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Explorationcited by 15
  31. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
  32. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  33. 2025Mano Technical Reportcited by 13
  34. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  35. 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 12
  36. 2025GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learningcited by 11
  37. 2025Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agentcited by 11
  38. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
  2. 2026Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
  3. 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks
  4. 2026MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search
  5. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  6. 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
  7. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
  8. 2025Improving GUI Grounding with Explicit Position-to-Coordinate Mapping