mlsanityData to 7 Oct 2026

Against The Achilles' Heel: A Survey on Red Teaming for Generative Models

arXiv 2404.00629 ↗First posted 31 Mar 2024cs.CLcited by 55

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Generative models are rapidly gaining popularity and being integrated into everyday applications, raising concerns over their safe use as various vulnerabilities are exposed. In light of this, the field of red teaming is undergoing fast-paced growth, highlighting the need for a comprehensive survey covering the entire pipeline and addressing emerging topics. Our extensive survey, which examines over 120 papers, introduces a taxonomy of fine-grained attack strategies grounded in the inherent capabilities of language models. Additionally, we have developed the "searcher" framework to unify various automatic red teaming approaches. Moreover, our survey covers novel areas including multimodal attacks and defenses, risks around LLM-based agents, overkill of harmless queries, and the balance between harmlessness and helpfulness.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. First posted on arXiv 31 Mar 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 26; 2026: 18.

55 papers.

Sort:

By area:

  1. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  2. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  3. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  4. 2024garak: A Framework for Security Probing Large Language Modelscited by 48
  5. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  6. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  7. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  8. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 23
  9. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  10. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  11. 2024Arabic Dataset for LLM Safeguard Evaluationcited by 17
  12. 2026Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teamingcited by 15
38 more
  1. 2025Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Controlcited by 14
  2. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  3. 2025K2-Think: A Parameter-Efficient Reasoning Systemcited by 11
  4. 2025A Safety and Security Framework for Real-World Agentic Systemscited by 11
  5. 2024SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behaviorcited by 10
  6. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 10
  7. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  8. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 8
  9. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  10. 2024Hacc-Man: An Arcade Game for Jailbreaking LLMscited by 4
  11. 2025Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Settingcited by 3
  12. 2025Position: Beyond Assistance -- Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Carecited by 3
  13. 2025K2-V2: A 360-Open, Reasoning-Enhanced LLMcited by 3
  14. 2026Trusting What You Cannot See: Auditable Fine-Tuning and Inference for Proprietary AIcited by 3
  15. 2026Adaptive Instruction Composition for Automated LLM Red-Teamingcited by 3
  16. 2024RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreakingcited by 2
  17. 2024Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capabilitycited by 2
  18. 2025A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Roboticscited by 2
  19. 2025AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applicationscited by 2
  20. 2025Red Teaming Large Language Models for Healthcarecited by 1
  21. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  22. 2025VisualDAN: Exposing Vulnerabilities in VLMs with Visual-Driven DAN Commandscited by 1
  23. 2025N-GLARE: An Non-Generative Latent Representation-Efficient LLM Safety Evaluatorcited by 1
  24. 2026Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactionscited by 1
  25. 2026STAR-Teaming: A Strategy-Response Multiplex Network Approach to Automated LLM Red Teamingcited by 1
  26. 2026JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimizationcited by 1
  27. 2026Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Modelscited by 1
  28. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  29. 2025MGC: A Compiler Framework Exploiting Compositional Blindness in Aligned LLMs for Malware Generationcited by 0
  30. 2025Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreakingcited by 0
  31. 2025Let the Bees Find the Weak Spots: A Path Planning Perspective on Multi-Turn Jailbreak Attacks against LLMscited by 0
  32. 2025Are LLMs Good Safety Agents or a Propaganda Engine?cited by 0
  33. 2026STELP: Secure Transpilation and Execution of LLM-Generated Programscited by 0
  34. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0
  35. 2026RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studiescited by 0
  36. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuningcited by 0
  37. 2026Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigationcited by 0
  38. 2026Sch"utzen: Evaluating LLM Safety in Bulgarian and German Contextscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools
  2. 2025CoP: Agentic Red-teaming for Large Language Models using Composition of Principles
  3. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding
  4. 2026TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  6. 2026AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models
  7. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
  8. 2025Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints