# Detecting Safety Violations Across Many Agent Traces

arXiv: https://arxiv.org/abs/2604.11806 · HTML: https://mlsanity.com/paper/2604.11806/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 30 of 34)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [Prompt Injection Attack to Tool Selection in LLM Agents](https://mlsanity.com/paper/2504.19793/) (2025)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [Clio: Privacy-Preserving Insights into Real-World AI Use](https://mlsanity.com/paper/2412.13678/) (2024)
- [MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents](https://mlsanity.com/paper/2502.05174/) (2025)
- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [Can Language Models Solve Olympiad Programming?](https://mlsanity.com/paper/2404.10952/) (2024)
- [ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases](https://mlsanity.com/paper/2510.20270/) (2025)
- [From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows](https://mlsanity.com/paper/2506.23260/) (2025)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors](https://mlsanity.com/paper/2602.22755/) (2026)
- [Robust Online Monitoring of Signal Temporal Logic](https://mlsanity.com/paper/1506.08234/) (2015)
- [Reliable Weak-to-Strong Monitoring of LLM Agents](https://mlsanity.com/paper/2508.19461/) (2025)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats](https://mlsanity.com/paper/2411.17693/) (2024)
- [Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition](https://mlsanity.com/paper/2507.20526/) (2025)
- [Async Control: Stress-testing Asynchronous Control Measures for LLM Agents](https://mlsanity.com/paper/2512.13526/) (2025)
- [Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses](https://mlsanity.com/paper/2407.02551/) (2024)
- [Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis](https://mlsanity.com/paper/2601.20103/) (2026)
- [CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D](https://mlsanity.com/paper/2511.09904/) (2025)

## Built on by (most cited first, 14 of 14)

- [Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack](https://mlsanity.com/paper/2605.12673/) (2026)
- [Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops](https://mlsanity.com/paper/2606.08960/) (2026)
- [Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests](https://mlsanity.com/paper/2606.07379/) (2026)
- [BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure](https://mlsanity.com/paper/2609.11028/) (2026)
- [AuditRepairBench: A Paired-Execution Trace Corpus for Evaluator-Channel Ranking Instability in Agent Repair](https://mlsanity.com/paper/2605.04624/) (2026)
- [Computer Use at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2605.08261/) (2026)
- [Property-Level Reconstructability of Agent Decisions: An Anchor-Level Pilot Across Vendor SDK Adapter Regimes](https://mlsanity.com/paper/2605.12078/) (2026)
- [EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models](https://mlsanity.com/paper/2609.01611/) (2026)
- [The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability](https://mlsanity.com/paper/2609.07162/) (2026)
- [TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces](https://mlsanity.com/paper/2609.33295/) (2026)
- [AuraForge: Scaling Security Supervision for Training Coding Agents](https://mlsanity.com/paper/2610.00850/) (2026)
- [Towards a Unified Misuse Monitoring Benchmark](https://mlsanity.com/paper/2610.07089/) (2026)
- [Scalable Weak-to-Strong Verifier Hardening for Agent Environments](https://mlsanity.com/paper/2610.15924/) (2026)
- [ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty](https://mlsanity.com/paper/2610.17016/) (2026)

## Abstract (arXiv)

To identify safety violations, auditors often search over large sets of agent traces. This search is difficult because failures are often rare, complex, and sometimes even adversarially hidden and only detectable when multiple traces are analyzed together. These challenges arise in diverse settings such as misuse campaigns, covert sabotage, reward hacking, and prompt injection. Existing approaches struggle here for several reasons. Per-trace judges miss failures that only become visible across traces, naive agentic auditing does not scale to large trace collections, and fixed monitors are brittle to unanticipated behaviors. We introduce Meerkat, which combines clustering with agentic search to uncover violations specified in natural language. Through structured search and adaptive investigation of promising regions, Meerkat finds sparse failures without relying on seed scenarios, fixed workflows, or exhaustive enumeration. Across misuse, misalignment, and task gaming settings, Meerkat significantly improves detection of safety violations over baseline monitors, discovers widespread developer cheating on a top agent benchmark, and finds nearly 4x more examples of reward hacking on CyBench than previous audits.