# Auditing Agent Harness Safety

arXiv: https://arxiv.org/abs/2605.14271 · HTML: https://mlsanity.com/paper/2605.14271/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 14 May 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 28 of 28)

- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents](https://mlsanity.com/paper/2406.13352/) (2024)
- [ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents](https://mlsanity.com/paper/2410.06703/) (2024)
- [OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents](https://mlsanity.com/paper/2506.14866/) (2025)
- [Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents](https://mlsanity.com/paper/2604.06132/) (2026)
- [ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis](https://mlsanity.com/paper/2604.02022/) (2026)
- [ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces](https://mlsanity.com/paper/2604.05172/) (2026)
- [ClawBench: Can AI Agents Complete Everyday Online Tasks?](https://mlsanity.com/paper/2604.08523/) (2026)
- [Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats](https://mlsanity.com/paper/2603.11619/) (2026)
- [TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems](https://mlsanity.com/paper/2511.05269/) (2025)
- [AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems](https://mlsanity.com/paper/2602.11510/) (2026)
- [ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watchers](https://mlsanity.com/paper/2603.24414/) (2026)
- [Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw](https://mlsanity.com/paper/2604.04759/) (2026)
- [ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents](https://mlsanity.com/paper/2604.23781/) (2026)
- [A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)](https://mlsanity.com/paper/2602.14364/) (2026)
- [A Systematic Security Evaluation of OpenClaw and Its Variants](https://mlsanity.com/paper/2604.03131/) (2026)
- [Agent Audit: A Security Analysis System for LLM Agent Applications](https://mlsanity.com/paper/2603.22853/) (2026)
- [Quantifying Trust: Financial Risk Management for Trustworthy AI Agents](https://mlsanity.com/paper/2604.03976/) (2026)
- [Emergent Risks in Generative Multi-Agent Systems](https://mlsanity.com/paper/2603.27771/) (2026)
- [GroupGuard: A Framework for Modeling and Defending Collusive Attacks in Multi-Agent Systems](https://mlsanity.com/paper/2603.13940/) (2026)
- [Do Phone-Use Agents Respect Your Privacy?](https://mlsanity.com/paper/2604.00986/) (2026)

## Built on by (most cited first, 11 of 11)

- [SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment](https://mlsanity.com/paper/2609.02786/) (2026)
- [AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents](https://mlsanity.com/paper/2607.27294/) (2026)
- [HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety](https://mlsanity.com/paper/2608.17597/) (2026)
- [Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agents](https://mlsanity.com/paper/2608.08793/) (2026)
- [Reachability-Based Capability Confinement for LLM Agents under Indirect Prompt Injection](https://mlsanity.com/paper/2608.30041/) (2026)
- [SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems](https://mlsanity.com/paper/2609.00595/) (2026)
- [Trust the Brand, Lose Control: How Identity Hijacks LLM Agent Orchestration](https://mlsanity.com/paper/2609.32635/) (2026)
- [HARDE: Optimizing Agent Harnesses for Runtime Risk Detection and Execution Control](https://mlsanity.com/paper/2609.38291/) (2026)
- [CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents](https://mlsanity.com/paper/2610.08871/) (2026)
- [MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation](https://mlsanity.com/paper/2610.15714/) (2026)
- [Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents](https://mlsanity.com/paper/2610.20344/) (2026)

## Abstract (arXiv)

LLM agents increasingly run inside execution harnesses that dispatch tools, allocate resources, and route messages between specialized components. However, a harness can return a correct, benign answer over a trajectory that accesses unauthorized resources or leaks context to the wrong agent. Output-level evaluation cannot see these failures, yet most safety benchmarks score only final outputs or terminal states, even though many violations occur mid-trajectory rather than at termination. The central question is whether the harness respects user intent, permission boundaries, and information-flow constraints throughout execution. To address this gap, we propose HarnessAudit, a framework that audits full execution trajectories across boundary compliance, execution fidelity, and system stability, with a focus on multi-agent harnesses where these risks are most pronounced. We further introduce HarnessAudit-Bench, a benchmark of 210 tasks across eight real-world domains, instantiated in both single-agent and multi-agent configurations with embedded safety constraints. Evaluating ten harness configurations across frontier models and three multi-agent frameworks, we find that: (i) task completion is misaligned with safe execution, and violations accumulate with trajectory length; (ii) safety risks vary across domains, task types, and agent roles; (iii) most violations concentrate in resource access and inter-agent information transfer; and (iv) multi-agent collaboration expands the safety risk surface, while harness design sets the upper bound of safe deployment.