# WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

arXiv: https://arxiv.org/abs/2605.10912 · HTML: https://mlsanity.com/paper/2605.10912/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 35)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)

## Built on by (most cited first, 30 of 37)

- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.28424/) (2026)
- [Agent Skill Evaluation and Evolution: Frameworks and Benchmarks](https://mlsanity.com/paper/2606.11435/) (2026)
- [Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks](https://mlsanity.com/paper/2606.12344/) (2026)
- [MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents](https://mlsanity.com/paper/2606.16748/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI](https://mlsanity.com/paper/2607.22368/) (2026)
- [From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems](https://mlsanity.com/paper/2608.15127/) (2026)
- [SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents](https://mlsanity.com/paper/2606.16591/) (2026)
- [Intern-S2-Preview: Scientific Agentic Foundation Model](https://mlsanity.com/paper/2608.13505/) (2026)
- [RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions](https://mlsanity.com/paper/2606.03889/) (2026)
- [Occamy-1.0: Open Pareto-frontier 35B Intelligence for Co-work](https://mlsanity.com/paper/2609.11977/) (2026)
- [EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions](https://mlsanity.com/paper/2606.23654/) (2026)
- [HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety](https://mlsanity.com/paper/2608.17597/) (2026)
- [Qwen3.8-Omni: Towards Native Omni-Modal Agents](https://mlsanity.com/paper/2609.25611/) (2026)
- [Enhancing LLM Metacognition via Cognitive Pairwise Training](https://mlsanity.com/paper/2606.00869/) (2026)
- [World-Model Collapse as a Phase Transition](https://mlsanity.com/paper/2606.31399/) (2026)
- [PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows](https://mlsanity.com/paper/2607.06008/) (2026)
- [ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents](https://mlsanity.com/paper/2607.28037/) (2026)
- [Terminal Agents: A Survey of AI Agents in Command-Line Environments](https://mlsanity.com/paper/2608.20485/) (2026)
- [Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems](https://mlsanity.com/paper/2605.27492/) (2026)
- [Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games](https://mlsanity.com/paper/2606.19338/) (2026)
- [Just A Rather Very Intelligent Spoken Agent](https://mlsanity.com/paper/2607.16610/) (2026)
- [AgentDecarbonizer: Carbon-Aware Execution for AI Agents](https://mlsanity.com/paper/2608.20566/) (2026)
- [Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents](https://mlsanity.com/paper/2609.01487/) (2026)
- [SkillLift: Learning Dense Rubrics from Sparse Oracles for Efficient Skill Evolution](https://mlsanity.com/paper/2609.15396/) (2026)
- [TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces](https://mlsanity.com/paper/2609.33295/) (2026)
- [CompoWorld: Compositional Environment Scaling for General Agents](https://mlsanity.com/paper/2609.33665/) (2026)
- [Environment Steering: Using Data Flow Control to Improve Agent Utility and Safety](https://mlsanity.com/paper/2609.35807/) (2026)
- [Talk2Agent: Benchmarking Voice Interfaces for Text Agents](https://mlsanity.com/paper/2609.38867/) (2026)

## Abstract (arXiv)

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.