# ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

arXiv: https://arxiv.org/abs/2604.05172 · HTML: https://mlsanity.com/paper/2604.05172/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 46 of its 71 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 46)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents](https://mlsanity.com/paper/2410.06703/) (2024)
- [NATURAL PLAN: Benchmarking LLMs on Natural Language Planning](https://mlsanity.com/paper/2406.04520/) (2024)
- [Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward](https://mlsanity.com/paper/2602.12430/) (2026)
- [OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents](https://mlsanity.com/paper/2506.14866/) (2025)
- [Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale](https://mlsanity.com/paper/2601.10338/) (2026)
- [WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks](https://mlsanity.com/paper/2407.05291/) (2024)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments](https://mlsanity.com/paper/2411.02305/) (2024)
- [ACEBench: Who Wins the Match Point in Tool Usage?](https://mlsanity.com/paper/2501.12851/) (2025)
- [WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting](https://mlsanity.com/paper/2405.00823/) (2024)
- [MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers](https://mlsanity.com/paper/2602.00933/) (2026)

## Built on by (most cited first, 30 of 33)

- [Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies](https://mlsanity.com/paper/2605.03596/) (2026)
- [ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents](https://mlsanity.com/paper/2604.23781/) (2026)
- [Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack](https://mlsanity.com/paper/2605.12673/) (2026)
- [Auditing Agent Harness Safety](https://mlsanity.com/paper/2605.14271/) (2026)
- [ClawEnvKit: Automatic Environment Generation for Claw-Like Agents](https://mlsanity.com/paper/2604.18543/) (2026)
- [Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks](https://mlsanity.com/paper/2606.12344/) (2026)
- [π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows](https://mlsanity.com/paper/2605.14678/) (2026)
- [Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction](https://mlsanity.com/paper/2607.20911/) (2026)
- [LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments](https://mlsanity.com/paper/2605.10779/) (2026)
- [ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents](https://mlsanity.com/paper/2605.14133/) (2026)
- [SentinelBench: A Benchmark for Long-Running Monitoring Agents](https://mlsanity.com/paper/2606.05342/) (2026)
- [OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills](https://mlsanity.com/paper/2607.20121/) (2026)
- [Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values](https://mlsanity.com/paper/2605.10365/) (2026)
- [ASEval: Automated Trajectory-Level Security Testing for Autonomous Agents](https://mlsanity.com/paper/2605.22321/) (2026)
- [STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios](https://mlsanity.com/paper/2606.10394/) (2026)
- [HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety](https://mlsanity.com/paper/2608.17597/) (2026)
- [PalmClaw: A Native On-Device Agent Framework for Mobile Phones](https://mlsanity.com/paper/2607.13027/) (2026)
- [ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents](https://mlsanity.com/paper/2607.28037/) (2026)
- [What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files](https://mlsanity.com/paper/2608.08453/) (2026)
- [Terminal Agents: A Survey of AI Agents in Command-Line Environments](https://mlsanity.com/paper/2608.20485/) (2026)
- [DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows](https://mlsanity.com/paper/2608.26546/) (2026)
- [Influence Is Not Authority: When Causal Guardrail Signals Make Legitimate Tool Use Look Like an Attack in Tool-Using LLM Agents](https://mlsanity.com/paper/2608.29942/) (2026)
- [BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure](https://mlsanity.com/paper/2609.11028/) (2026)
- [ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts](https://mlsanity.com/paper/2608.22510/) (2026)
- [DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents](https://mlsanity.com/paper/2609.06059/) (2026)
- [SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops](https://mlsanity.com/paper/2609.19169/) (2026)
- [Et Tu, Brute? Economic Misalignment in Personal AI Agents](https://mlsanity.com/paper/2609.24927/) (2026)
- [The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge](https://mlsanity.com/paper/2609.30604/) (2026)
- [EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks](https://mlsanity.com/paper/2609.31906/) (2026)
- [Trust and Task Completion in the World of Consumer AI Agents](https://mlsanity.com/paper/2609.33017/) (2026)

## Abstract (arXiv)

Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification. We release the trajectories and future dataset at https://clawsbench.com.