mlsanityData to 7 Oct 2026

cotomi Act: Learning to Automate Work by Watching You

arXiv 2605.03231 ↗First posted 4 May 2026cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

What if a browser agent could learn your work simply by watching you do it? We present cotomi Act, a browser-based computer-using agent that combines reliable multi-step task execution with persistent organizational knowledge learned from user behavior. For execution, an agent scaffold with adaptive lazy observation, verbal-diff-based history compression, coarse-grained actions, and test-time scaling via best-of-N action selection achieves 80.4% on the 179-task WebArena human-evaluation subset, exceeding the reported 78.2% human baseline. For organizational knowledge, a behavior-to-knowledge pipeline passively observes the user's browsing and progressively abstracts it into artifacts (task boards, wiki) exposed through a shared workspace editable by both user and agent. A controlled proxy evaluation confirms that task success improves as behavior-derived knowledge accumulates. In our live demonstration, attendees interact with the system in a real browser, issuing tasks and observing end-to-end autonomous execution and shared knowledge management.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

15 of 32 references.

References · 15

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  6. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  7. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  8. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  9. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  10. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
  11. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  12. 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 14
3 more
  1. 2026OpAgent: Operator Agent for Web Navigationcited by 8
  2. 2025From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Productioncited by 5
  3. 2026Read More, Think More: Revisiting Observation Reduction for Web Agentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge
  2. 2025UI-Evol: Automatic Knowledge Evolving for Computer Use Agents
  3. 2026KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
  4. 2026Token Efficient Task Execution via Application Behavior Modeling for Web Agents
  5. 2026Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems
  6. 2026Agent-Computer Observation Interfaces Enable Dynamic Computer Use
  7. 2025Learning from Online Videos at Inference Time for Computer-Use Agents
  8. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents