mlsanityData to 14 Oct 2026

Benchmarking Misuse Mitigation Against Covert Adversaries

arXiv 2506.06414 ↗First posted 6 Jun 2025cs.CR cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Existing language model safety evaluations focus on overt attacks and low-stakes tasks. In reality, an attacker can easily subvert existing safeguards by requesting help on small, benign-seeming tasks across many independent queries. Because the individual queries do not appear harmful, the attack is hard to detect. However, when combined, these fragments uplift misuse by helping the attacker complete hard and dangerous tasks. Toward identifying defenses against such strategies, we develop Benchmarks for Stateful Defenses (BSD), a data generation pipeline that automates evaluations of covert attacks and corresponding defenses. Using this pipeline, we curate two new datasets that are consistently refused by frontier models and are too difficult for weaker open-weight models. This enables us to evaluate decomposition attacks, which are found to be effective misuse enablers, and to highlight stateful defenses as a promising countermeasure.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 55 of its 81 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

55 of 81 references.

References · 55

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  12. 2022Red Teaming Language Models with Language Modelscited by 1,026
38 more
  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  10. 2024A StrongREJECT for Empty Jailbreakscited by 385
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  12. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  13. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  14. 2023Model evaluation for extreme riskscited by 177
  15. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  16. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 165
  17. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
  18. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  19. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  20. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  21. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  22. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  23. 2024Rule Based Rewards for Language Model Safetycited by 108
  24. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 101
  25. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  26. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  27. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 72
  28. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  29. 2019Stateful Detection of Black-Box Adversarial Attackscited by 56
  30. 2025Do Large Language Model Benchmarks Test Reliability?cited by 54
  31. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  32. 2024Obfuscated Activations Bypass LLM Latent-Space Defensescited by 45
  33. 2020Blacklight: Scalable Defense for Neural Networks against Query-Based Black-Box Attackscited by 37
  34. 2023SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Securitycited by 37
  35. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  36. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 36
  37. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  38. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 31

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2025Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testingcited by 15
  2. 2026Detecting Safety Violations Across Many Agent Tracescited by 14
  3. 2026Training Agents to Self-Report Misbehaviorcited by 6
  4. 2026Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectorscited by 1
  5. 2026A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Modelscited by 1
  6. 2026One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intentcited by 0
  7. 2026Stateful Online Monitoring Catches Distributed Agent Attackscited by 0
  8. 2026Cross-Session Decomposition Attacks: Scaling Risk and Intent-Aligned Retrieval Defensecited by 0