# Which Defense Closes Which Threat? Attributing OWASP-LLM-Top-10 Coverage and Its Brittleness Under Paraphrasing

arXiv: https://arxiv.org/abs/2606.02822 · HTML: https://mlsanity.com/paper/2606.02822/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 9 of 9)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [ACE: A Security Architecture for LLM-Integrated App Systems](https://mlsanity.com/paper/2504.20984/) (2025)
- [garak: A Framework for Security Probing Large Language Models](https://mlsanity.com/paper/2406.11036/) (2024)
- [Mitigating the OWASP Top 10 For Large Language Models Applications using Intelligent Agents](https://mlsanity.com/paper/2601.18105/) (2026)
- [Countermind: A Multi-Layered Security Architecture for Large Language Models](https://mlsanity.com/paper/2510.11837/) (2025)
- [Is the OWASP Top 10 list comprehensive enough for writing secure code?](https://mlsanity.com/paper/2002.11269/) (2020)

## Built on by (most cited first, 2 of 2)

- [From Attack Simulation to SIEM Rule: Deterministic Detection-as-Code Synthesis with Probe-Level Traceability](https://mlsanity.com/paper/2606.05252/) (2026)
- [Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation](https://mlsanity.com/paper/2607.20494/) (2026)

## Abstract (arXiv)

Production LLM applications stack several defense families -- refusal-phrase filters, token-budget controls, model allowlists, rate limits, tool-registry authentication -- yet existing breach-and-attack-simulation (BAS) benchmarks report a single aggregate coverage number, hiding which family closes which threat. We measure attribution. We add four OWASP-LLM-Top-10-aware agents to a 21-agent baseline scanner and target a lattice of four synthetic LLM endpoints: L0 (no defenses), L1 (refusal-only), L2 (budget-only), and L3 (full stack). L1 and L2 are sibling single-axis ablations, not subsets of each other; L3 is their union plus tool-registry authentication and credential scrubbing. Across N=10 replications, the per-OWASP finding count is clean: refusal alone removes all LLM01 (jailbreak) and LLM07 (system-prompt leakage) findings; budget alone removes all LLM02 (sensitive-info disclosure) and LLM10 (unbounded consumption) findings by terminating multi-step sequences; LLM06 (excessive agency) requires the full stack. We probe brittleness under paraphrasing: with 300 Gemini-generated paraphrases (K=5 over a 60-template brittleness corpus), L1 refusal block rate falls 15 pp on LLM01 and 25 pp on LLM07. A fifth target, L4-real, swaps the stub backend for Gemini-2.5-flash behind the same L3 regex and matches L1 exactly, indicating no measurable alignment contribution beyond the regex (not a general claim about alignment). Budget controls show no drop (0 pp once the rate-limit floor is factored out). A refusal whitelist that clears a static benchmark can be defeated by an LLM-driven paraphraser without changing attack intent; a budget control resists the same mutation.