mlsanityData to 14 Oct 2026

STACK: Adversarial Attacks on LLM Safeguard Pipelines

arXiv 2506.24068 ↗First posted 30 Jun 2025cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic and OpenAI guard their latest Opus 4 model and GPT-5 models using such defense pipelines, and other frontier developers including Google DeepMind pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

37 of 62 references.

References · 37

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  10. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  11. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
25 more
  1. 2024A StrongREJECT for Empty Jailbreakscited by 385
  2. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  4. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  5. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  6. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  7. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  9. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  10. 2023An Overview of Catastrophic AI Riskscited by 222
  11. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  12. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  13. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  14. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  15. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 88
  16. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  17. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 61
  18. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 52
  19. 2024Obfuscated Activations Bypass LLM Latent-Space Defensescited by 45
  20. 2023Exploiting Novel GPT-4 APIscited by 28
  21. 2024Scaling Trends in Language Model Robustnesscited by 20
  22. 2025DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimizationcited by 12
  23. 2024Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approachcited by 6
  24. 2025Peering Behind the Shield: Guardrail Identification in Large Language Modelscited by 5
  25. 2025Effective Black-Box Multi-Faceted Attacks Breach Vision Large Language Model Guardrailscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  2. 2025SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAGcited by 13
  3. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 11
  4. 2026Boundary Point Jailbreaking of Black-Box LLMscited by 8
  5. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systemscited by 4
  6. 2026Systematic Scaling Analysis of Jailbreak Attacks in Large Language Modelscited by 2
  7. 2026Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demandscited by 2
  8. 2026Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMscited by 1
  9. 2026AI Security Leaderboard: Methodology, Results and Minimal Standardcited by 1
  10. 2026Directional Embedding Smoothing for Robust Vision Language Modelscited by 0
  11. 2026On the Hardness of Junking LLMscited by 0
  12. 2026A Systematic Investigation of RL-Jailbreaking in LLMscited by 0
2 more
  1. 2026Sharding Prevents LLM Oversight Failures and Adversarial Exploitationcited by 0
  2. 2026Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025JavelinGuard: Low-Cost Transformer Architectures for LLM Security
  2. 2025Adversarial Suffix Filtering: a Defense Pipeline for LLMs
  3. 2025Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
  4. 2026Reflect-Guard: Enhancing LLM Safeguards against Adversarial Prompts via Logical Self-Reflection
  5. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks
  6. 2026Cascade: Composing Software-Hardware Attack Gadgets for Adversarial Threat Amplification in Compound AI Systems
  7. 2026Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs
  8. 2026Low-ASR Backdoors: Exploiting Attack Success Rate Reduction and Attacker-Defender Asymmetry ↗