Data to 14 Oct 2026

TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents

arXiv 2504.12679 ↗First posted 17 Apr 2025cs.CVcited by 42

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Building Graphical User Interface (GUI) agents is a promising research direction, which simulates human interaction with computers or mobile phones to perform diverse GUI tasks. However, a major challenge in developing generalized GUI agents is the lack of sufficient trajectory data across various operating systems and applications, mainly due to the high cost of manual annotations. In this paper, we propose the TongUI framework that builds generalized GUI agents by learning from rich multimodal web tutorials. Concretely, we crawl and process online GUI tutorials (such as videos and articles) into GUI agent trajectory data, through which we produce the GUI-Net dataset containing 143K trajectory data across five operating systems and more than 200 applications. We develop the TongUI agent by fine-tuning Qwen2.5-VL-3B/7B models on GUI-Net, which show remarkable performance improvements on commonly used grounding and navigation benchmarks, outperforming baseline agents about 10% on multiple benchmarks, showing the effectiveness of the GUI-Net dataset and underscoring the significance of our TongUI framework. We will fully open-source the code, the GUI-Net dataset, and the trained models soon.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 48 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

33 of 48 references.

Closest ancestors · 10

Same direction, newest first

Other references · 23

2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148

11 more

2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 95
2022A data-driven approach for learning to control computerscited by 88
2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 51
2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
2024AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistantcited by 27
2024E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTioncited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 23.

42 papers.

Sort:

By area:

2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 26
2025A Survey on Agentic Multimodal Large Language Modelscited by 23
2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
2025WALT: Web Agents that Learn Toolscited by 16
2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 15

30 more

2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
2025Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agentscited by 11
2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 11
2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learningcited by 7
2025AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purificationcited by 6
2026Enhancing Web Agents with a Hierarchical Memory Treecited by 6
2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agentscited by 5
2026GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotationcited by 4
2025Universal Retrieval for Multimodal Trajectory Modelingcited by 3
2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
2025GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Taskscited by 3
2026Interactive Reward Agent: GUI Task Evaluation via Environment-State Verificationcited by 3
2025Learning from Online Videos at Inference Time for Computer-Use Agentscited by 2
2026Benchmarking and Improving GUI Agents in High-Dynamic Environmentscited by 2
2026MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?cited by 2
2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmarkcited by 1
2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
2026DocOS: Towards Proactive Document-Guided Actions in GUI Agentscited by 1
2026CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editingcited by 1
2026PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agentscited by 1
2026SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesiscited by 1
2026PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editorscited by 0
2026Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perceptioncited by 0
2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
2026Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extractioncited by 0
2026SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agentscited by 0
2026Selective Knowledge Control for Continual GUI Agent Learning over Application Streamscited by 0
2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
2026Adaptive-Group Process Rewards for GUI Agentscited by 0

Similar, not cited

Close in wording, no citation either way