mlsanityData to 14 Oct 2026

Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes

arXiv 2510.04452 ↗First posted 6 Oct 2025cs.HC cs.AIcited by 2

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands. Current research focuses on training and evaluating the underlying models, leaving these agents' user experience (UX) understudied. We conducted two studies to understand the design space of agent UX (RQ1) and the support required to prototype it (RQ2). First, we develop a taxonomy of design considerations for agent UX, comprising 21 subcategories of UX considerations. Then, through a requirements elicitation study with 12 participants---including six agent experts---we identify five Activities and six Desired Capabilities needed in tools prototyping agent UX. Informed by these results, we created AgentUXlab, a design probe that enables developers to design agents with different UX approaches for a website and evaluate those experiences by executing prototypes in a browser. From a user study with 14 participants, we elucidate tooling insights and derive design implications for agent UX prototyping tools.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 85 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

34 of 85 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  2. 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
  3. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts11 Oct
  4. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  5. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb
  6. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan

Other references · 27

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  2. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  3. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  5. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  6. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  7. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  8. 2023Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfacescited by 72
  9. 2025Interactive Debugging and Steering of Multi-Agent AI Systemscited by 68
  10. 2024Large Language Models Empowered Personalized Web Agentscited by 66
  11. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  12. 2024Design Principles for Generative AI Applicationscited by 57
15 more
  1. 2024"We Need Structured Output": Towards User-centered Constraints on Large Language Model Outputcited by 46
  2. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 39
  3. 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 36
  4. 2025Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Supportcited by 34
  5. 2024Supporting Sensemaking of Large Language Model Outputs at Scalecited by 27
  6. 2019Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrationscited by 26
  7. 2024Prompts Are Programs Too! Understanding How Developers Build Software Containing Promptscited by 19
  8. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  9. 2024Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teamscited by 16
  10. 2023PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers' Workflowscited by 8
  11. 2024Development in times of hype: How freelancers explore Generative AI?cited by 8
  12. 2025TableTalk: Scaffolding Spreadsheet Development with a Language Agentcited by 5
  13. 2025FlowForge: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffoldcited by 3
  14. 2024Orbit: A Framework for Designing and Evaluating Multi-objective Rankerscited by 1
  15. 2025Understanding Prompt Programming Tasks and Questionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 19
  2. 2026Mapping the Design Space of User Experience for Computer Use Agentscited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2026A Framework of User Experience Principles for Human-AI Agent Interaction in the Workplace
  2. 2026From SERPs to Agents: A Platform for Comparative Studies of Information Interaction
  3. 2026Affora: A Design System for Agent-Friendly Interfaces
  4. 2026UXCascade: Scalable Usability Testing with Simulated User Agents
  5. 2026Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents
  6. 2025Computer-Use Agents as Judges for Generative User Interface
  7. 2025Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies
  8. 2026The Tool Illusion: Rethinking Tool Use in Web Agents