mlsanityData to 7 Oct 2026

ProCUA-SFT Technical Report

arXiv 2606.17321 ↗First posted 15 Jun 2026cs.LG cs.CVcited by 2

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Training computer-use agents (CUAs) -- models that interact with graphical desktops through screenshots and keyboard/mouse actions -- requires large-scale, diverse trajectory data collected in full desktop environments. The largest public resource, AgentNet (22.5K human trajectories), leads to negative transfer when used for supervised fine-tuning (SFT): continuing training UI-TARS 7B on AgentNet causes OSWorld success rate to fall from 26.3% to 8-10%. We present ProCUA-SFT, a dataset of 3.1M step-level SFT samples distilled from 93K synthetic trajectories across 2,484 application combinations. The dataset is produced by a fully automated pipeline that (i) synthesizes grounded tasks on live desktops seeded with real-world content -- 912 spreadsheets from SpreadsheetBench, approximately 10K permissively-licensed presentations from Zenodo10K, and multi-application OSWorld configs -- and (ii) verifies each task's feasibility through binary precondition checking before rollout. A single VLM (Kimi-K2.5) serves as goal generator, precondition judge, and trajectory executor, eliminating planner-actor capability gaps. Each trajectory is expanded into step-prefix samples that exactly reproduce the context layout seen at inference time. Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields 45.0% on OSWorld -- an 18.7 percentage-point improvement over the base model and over 35% above AgentNet-trained counterparts. A subset of ProCUA was incorporated into the training data for the Nemotron 3 Nano Omni model, contributing to its computer-use capabilities.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 35 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

30 of 35 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
  2. 2025Efficient Agent Training for Computer Use20 May
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  4. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  6. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  7. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  10. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 20

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  6. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  7. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  8. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  9. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  10. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  11. 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 100
  12. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
8 more
  1. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  2. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuningcited by 51
  3. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 41
  4. 2024ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Datacited by 39
  5. 2025Go-Browse: Training Web Agents with Structured Explorationcited by 32
  6. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31
  7. 2024PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital Worldcited by 12
  8. 2025SynthAgent: Adapting Web Agents with Synthetic Supervisioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Grounding Computer Use Agents on Human Demonstrations
  2. 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents
  3. 2026OSWorld-Pro: Process-based Evaluation for Computer Use Agents
  4. 2025Scaling Agents for Computer Use
  5. 2026Qwen-CUA: Native Computer Use for (almost) Everything
  6. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
  7. 2026PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning
  8. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents