Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex
Not on the map. Why?
Abstract (arXiv)
As agent systems move into increasingly diverse execution settings, trajectory-level safety evaluation and diagnosis require benchmarks that evolve with them. ATBench is a diverse and realistic agent trajectory benchmark for safety evaluation and diagnosis. This report presents ATBench-Claw and ATBench-Codex, two domain-customized extensions that carry ATBench into the OpenClaw and OpenAI Codex / Codex-runtime settings. The key adaptation mechanism is to analyze each new setting, customize the three-dimensional Safety Taxonomy over risk source, failure mode, and real-world harm, and then use that customized taxonomy to define the benchmark specification consumed by the shared ATBench construction pipeline. This extensibility matters because agent frameworks remain relatively stable at the architectural level even as their concrete execution settings, tool ecosystems, and product capabilities evolve quickly. Concretely, ATBench-Claw targets OpenClaw-sensitive execution chains over tools, skills, sessions, and external actions, while ATBench-Codex targets trajectories in the OpenAI Codex / Codex-runtime setting over repositories, shells, patches, dependencies, approvals, and runtime policy boundaries. Our emphasis therefore falls on taxonomy customization, domain-specific risk coverage, and benchmark design under a shared ATBench generation framework.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 7 of its 16 references that are in the map. First posted on arXiv 16 Apr 2026.
Builds on
7 of 16 references.
References · 7
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
- 2025Qwen3Guard Technical Reportcited by 155
- 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 151
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 8
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
- 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
- 2026RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessionscited by 3
- 2026HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnessescited by 3
- 2026SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agentscited by 1
- 2026HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses7 Aug 2026
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI12 Jun 2026
- 2026RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions2 Jun 2026
- 2026SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents1 Jun 2026
- 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security28 May 2026
- 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety21 May 2026