mlsanityData to 7 Oct 2026

Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI

arXiv 2409.15398 ↗First posted 23 Sep 2024cs.CR cs.AI cs.LGcited by 14

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

As generative AI, particularly large language models (LLMs), become increasingly integrated into production applications, new attack surfaces and vulnerabilities emerge and put a focus on adversarial threats in natural language and multi-modal systems. Red-teaming has gained importance in proactively identifying weaknesses in these systems, while blue-teaming works to protect against such adversarial attacks. Despite growing academic interest in adversarial risks for generative AI, there is limited guidance tailored for practitioners to assess and mitigate these challenges in real-world environments. To address this, our contributions include: (1) a practical examination of red- and blue-teaming strategies for securing generative AI, (2) identification of key challenges and open questions in defense development and evaluation, and (3) the Attack Atlas, an intuitive framework that brings a practical approach to analyzing single-turn input attacks, placing it at the forefront for practitioners. This work aims to bridge the gap between academic insights and practical security measures for the protection of generative AI systems.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 62 of its 78 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

62 of 78 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Exploring Straightforward Conversational Red-Teaming7 Sep
  2. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul
  3. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  4. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models19 Apr
  5. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  7. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications14 Nov
  8. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Models19 Oct
  9. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?3 Aug
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 52

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  12. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
38 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  4. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  5. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  8. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  9. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  10. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  11. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  12. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  13. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  14. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  15. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  16. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  17. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  18. 2020Clean-Label Backdoor Attacks on Video Recognition Modelscited by 162
  19. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  20. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  21. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  22. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  23. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  24. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 67
  25. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  26. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 57
  27. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Modelscited by 57
  28. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  29. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  30. 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 47
  31. 2024Current state of LLM Risks and AI Guardrailscited by 44
  32. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Responsecited by 28
  33. 2023Unveiling Safety Vulnerabilities of Large Language Modelscited by 26
  34. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 26
  35. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?cited by 25
  36. 2024Language Models in Dialogue: Conversational Maxims for Human-AI Interactionscited by 21
  37. 2024A Causal Explainable Guardrails for Large Language Modelscited by 18
  38. 2024Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitationscited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  2. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 12
  3. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  4. 2025Activated LoRA: Fine-tuned LLMs for Intrinsicscited by 9
  5. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  6. 2024Hey GPT, Can You be More Racist? Analysis from Crowdsourced Attempts to Elicit Biased Content from Generative AIcited by 6
  7. 2025Red Teaming AI Red Teamingcited by 3
  8. 2026Adaptive Instruction Composition for Automated LLM Red-Teamingcited by 3
  9. 2026Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Modelscited by 2
  10. 2025From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Modelscited by 0
  11. 2025BlackIce: A Containerized Red Teaming Toolkit for AI Security Testingcited by 0
  12. 2026Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarincited by 0
2 more
  1. 2026LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interfacecited by 0
  2. 2026Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigationcited by 0