cotomi Act: Learning to Automate Work by Watching You
Not on the map. Why?
Abstract (arXiv)
What if a browser agent could learn your work simply by watching you do it? We present cotomi Act, a browser-based computer-using agent that combines reliable multi-step task execution with persistent organizational knowledge learned from user behavior. For execution, an agent scaffold with adaptive lazy observation, verbal-diff-based history compression, coarse-grained actions, and test-time scaling via best-of-N action selection achieves 80.4% on the 179-task WebArena human-evaluation subset, exceeding the reported 78.2% human baseline. For organizational knowledge, a behavior-to-knowledge pipeline passively observes the user's browsing and progressively abstracts it into artifacts (task boards, wiki) exposed through a shared workspace editable by both user and agent. A controlled proxy evaluation confirms that task success improves as behavior-derived knowledge accumulates. In our live demonstration, attendees interact with the system in a real browser, issuing tasks and observing end-to-end autonomous execution and shared knowledge management.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 4 May 2026.
Builds on
15 of 32 references.
References · 15
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
- 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
- 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
- 2023SteP: Stacked LLM Policies for Web Actionscited by 66
- 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 14
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge
- 2025UI-Evol: Automatic Knowledge Evolving for Computer Use Agents
- 2026KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
- 2026Token Efficient Task Execution via Application Behavior Modeling for Web Agents
- 2026Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems
- 2026Agent-Computer Observation Interfaces Enable Dynamic Computer Use
- 2025Learning from Online Videos at Inference Time for Computer-Use Agents
- 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents