# CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation

arXiv: https://arxiv.org/abs/2501.16609 · HTML: https://mlsanity.com/paper/2501.16609/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 13 of 13)

- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning](https://mlsanity.com/paper/2406.11896/) (2024)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [Autonomous Evaluation and Refinement of Digital Agents](https://mlsanity.com/paper/2404.06474/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)

## Built on by (most cited first, 30 of 36)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [Magentic-UI: Towards Human-in-the-loop Agentic Systems](https://mlsanity.com/paper/2507.22358/) (2025)
- [BEARCUBS: A benchmark for computer-using web agents](https://mlsanity.com/paper/2503.07919/) (2025)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight](https://mlsanity.com/paper/2509.10723/) (2025)
- [VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents](https://mlsanity.com/paper/2509.07553/) (2025)
- [ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools](https://mlsanity.com/paper/2508.03284/) (2025)
- [Mapping the Design Space of User Experience for Computer Use Agents](https://mlsanity.com/paper/2602.07283/) (2026)
- [When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Tasks](https://mlsanity.com/paper/2510.05307/) (2025)
- [Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpages](https://mlsanity.com/paper/2505.22831/) (2025)
- [Modeling Distinct Human Interaction in Web Agents](https://mlsanity.com/paper/2602.17588/) (2026)
- [Revisable by Design: A Theory of Streaming LLM Agent Execution](https://mlsanity.com/paper/2604.23283/) (2026)
- [GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks](https://mlsanity.com/paper/2603.25864/) (2026)
- [Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents](https://mlsanity.com/paper/2509.12049/) (2025)
- [On the Regulatory Potential of User Interfaces for AI Agent Governance](https://mlsanity.com/paper/2512.00742/) (2025)
- [A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents](https://mlsanity.com/paper/2602.09310/) (2026)
- [Overseeing Agents Without Constant Oversight: Challenges and Opportunities](https://mlsanity.com/paper/2602.16844/) (2026)
- [SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows](https://mlsanity.com/paper/2606.08049/) (2026)
- [Plover: Steering GUI Agents through Plan-Centric Interaction](https://mlsanity.com/paper/2607.15193/) (2026)
- [Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes](https://mlsanity.com/paper/2510.04452/) (2025)
- [HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization](https://mlsanity.com/paper/2606.26614/) (2026)
- [Instruction Agent: Enhancing Agent with Expert Demonstration](https://mlsanity.com/paper/2509.07098/) (2025)
- [Designing Beyond Language: Sociotechnical Barriers in AI Health Technologies for Limited English Proficiency](https://mlsanity.com/paper/2511.07277/) (2025)
- [MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents](https://mlsanity.com/paper/2512.11147/) (2025)
- ["When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspaces](https://mlsanity.com/paper/2603.02050/) (2026)
- [WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks](https://mlsanity.com/paper/2604.06367/) (2026)
- [Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies](https://mlsanity.com/paper/2604.15607/) (2026)
- [Hierarchical Attacks for Multi-Modal Multi-Agent Reasoning](https://mlsanity.com/paper/2605.13213/) (2026)
- [CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks](https://mlsanity.com/paper/2606.09833/) (2026)

## Abstract (arXiv)

While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent's capabilities effectively. We propose CowPilot, a framework supporting autonomous as well as human-agent collaborative web navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform only 15.2% of the total steps. Even with human interventions during task execution, the agent successfully drives up to half of task success on its own. CowPilot can serve as a useful tool for data collection and agent evaluation across websites, which we believe will enable research in how users and agents can work together. Video demonstrations are available at https://oaishi.github.io/cowpilot.html