mlsanityData to 14 Oct 2026

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

arXiv 2608.15930 ↗First posted 16 Aug 2026cs.AI cs.CVcited by 1

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workflows rely on user-specific tools and tacit conventions, so unstated instructions can produce arbitrary variations across runs. We present UI-Mate, a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning. UI-Mate makes three contributions: A Scalable Environment-Grounded Training Stack: A closed-loop data engine automates task generation, environment construction, rollout, filtering, capability balancing, SFT, and online RL across massively parallel environments via unified task-verifier bundles. In-Context Demonstration Learning: A mechanism that transforms multimodal demonstrations into flexible subtask-level workflows, follows relevant demonstrated steps, and re-plans from the live interface. OSWorkerBench Benchmark and Insights: A benchmark of 100 long-horizon office tasks across 41 applications that supports instruction-only and demonstration-guided evaluation. Its demonstration resources separate a 33-task self-demo setting, built from successful strong-agent rollouts of the same targets, from a 45-task variant-demo setting, built from human recordings of related but non-identical tasks. Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena. On OSWorkerBench, it reaches 41.0% strict success and 76.9% progress, outperforming its Qwen3.6-27B base by 17.7 and 24.5 points. On the 33-task self-demo subset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, substantially improving long-horizon reliability. Project page: https://ui-mate.github.io.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 50 of its 57 references that are in the map. First posted on arXiv 16 Aug 2026.

Builds on

50 of 57 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents30 Jul
  2. 2026SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL13 Jul
  3. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks28 Jun
  4. 2026WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces8 Jun
  5. 2026Adaptive Milestone Reward for GUI Agents12 Feb
  6. 2026UI-Venus-1.5 Technical Report9 Feb
  7. 2026ShowUI-Aloha: Human-Taught GUI Agent12 Jan
  8. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
  9. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
  10. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
  11. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
  12. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
13 more
  1. 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
  2. 2025ProgRM: Build Better GUI Agents with Progress Rewards23 May
  3. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay22 May
  4. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  5. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
  6. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  7. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  8. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  9. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  10. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  12. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  13. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  8. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  9. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  10. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  11. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  12. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
13 more
  1. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 135
  2. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 77
  3. 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 70
  4. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 53
  5. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  6. 2025UFO2: The Desktop AgentOScited by 41
  7. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 35
  8. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 33
  9. 2025Single-stream Policy Optimizationcited by 28
  10. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 27
  11. 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 23
  12. 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Taskscited by 6
  13. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
  2. 2026Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
  3. 2025Instruction Agent: Enhancing Agent with Expert Demonstration
  4. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
  5. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
  6. 2026ANCHOR: Branch-Point Data Generation for GUI Agents
  7. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
  8. 2024Aria-UI: Visual Grounding for GUI Instructions