# Comparing Human Oversight Strategies for Computer-Use Agents

arXiv: https://arxiv.org/abs/2604.04918 · HTML: https://mlsanity.com/paper/2604.04918/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 64 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 18 of 18)

- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections](https://mlsanity.com/paper/2504.11281/) (2025)
- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [Magentic-UI: Towards Human-in-the-loop Agentic Systems](https://mlsanity.com/paper/2507.22358/) (2025)
- [Cocoa: Co-Planning and Co-Execution with AI Agents](https://mlsanity.com/paper/2412.10999/) (2024)
- [Intelligent AI Delegation](https://mlsanity.com/paper/2602.11865/) (2026)
- [Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight](https://mlsanity.com/paper/2509.10723/) (2025)
- [DECEPTICON: How Dark Patterns Manipulate Web Agents](https://mlsanity.com/paper/2512.22894/) (2025)
- [An Ontology of Dark Patterns Knowledge: Foundations, Definitions, and a Pathway for Shared Knowledge-Building](https://mlsanity.com/paper/2309.09640/) (2023)
- [Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent](https://mlsanity.com/paper/2411.01344/) (2024)
- [Optimizing Agent Planning for Security and Autonomy](https://mlsanity.com/paper/2602.11416/) (2026)
- [PrivWeb: Unobtrusive and Content-aware Privacy Protection For Web Agents](https://mlsanity.com/paper/2509.11939/) (2025)
- [Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsing](https://mlsanity.com/paper/2505.09875/) (2025)
- [SPARK: Real-Time Monitoring of Multi-Faceted Programming Exercises](https://mlsanity.com/paper/2601.22256/) (2026)
- [Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration](https://mlsanity.com/paper/2509.18008/) (2025)
- [From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agents](https://mlsanity.com/paper/2602.05987/) (2026)

## Built on by (most cited first, 6 of 6)

- [AI Agents Push Humans Out of the Loop](https://mlsanity.com/paper/2608.23642/) (2026)
- [Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agents](https://mlsanity.com/paper/2609.12205/) (2026)
- [Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization](https://mlsanity.com/paper/2604.27996/) (2026)
- [AI Snitches Get Glitches: Towards Evading Agentic Surveillance](https://mlsanity.com/paper/2606.25836/) (2026)
- [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026)
- [The Work Behind Delegation: A Framework for Supervising AI Coding Agents](https://mlsanity.com/paper/2609.24234/) (2026)

## Abstract (arXiv)

LLM-powered computer-use agents (CUAs) shift users from direct manipulation to supervising an unfolding action sequence, yet existing oversight research mostly evaluates static decisions rather than real-time oversight across an agent trajectory. We compared four oversight strategies with 48 participants across 192 live web sessions. Quantitative results show that oversight strategy affected exposure to problematic actions, but not users' ability to intervene. Behavioral analysis shows why: control often returned too early or too late, creating a timing mismatch within the action sequence. Even when control arrived at the right moment, users judged whether the agent acted correctly, not whether the action was safe. Front-loading oversight into a plan did not solve this either: plans constrained unplanned actions but left unlisted safeguards unaddressed, while upfront approval inhibited runtime scrutiny. These findings show that oversight depends less on maximizing control than on aligning authority, timing, and attention with decision-critical moments.