# CocoaBench: Evaluating Unified Digital Agents in the Wild

arXiv: https://arxiv.org/abs/2604.11201 · HTML: https://mlsanity.com/paper/2604.11201/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 12 of 12)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [OpenCUA: Open Foundations for Computer-Use Agents](https://mlsanity.com/paper/2508.09123/) (2025)
- [GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks](https://mlsanity.com/paper/2510.04374/) (2025)
- [The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution](https://mlsanity.com/paper/2510.25726/) (2025)
- [MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers](https://mlsanity.com/paper/2602.00933/) (2026)

## Built on by (most cited first, 6 of 6)

- [LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues](https://mlsanity.com/paper/2605.12493/) (2026)
- [WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces](https://mlsanity.com/paper/2606.09426/) (2026)
- [Masking Stale Observations Helps Search Agents -- Until It Doesn't: A Regime Map and Its Mechanism](https://mlsanity.com/paper/2606.00408/) (2026)
- [SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?](https://mlsanity.com/paper/2605.15777/) (2026)
- [PhoneBuddy: Training Open Models for Agentic Phone Use](https://mlsanity.com/paper/2606.23049/) (2026)
- [SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response](https://mlsanity.com/paper/2607.26791/) (2026)

## Abstract (arXiv)

LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.