mlsanityData to 7 Oct 2026

Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming

arXiv 2311.06237 ↗First posted 10 Nov 2023cs.CL cs.CR cs.HCcited by 20

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Engaging in the deliberate generation of abnormal outputs from Large Language Models (LLMs) by attacking them is a novel human activity. This paper presents a thorough exposition of how and why people perform such attacks, defining LLM red-teaming based on extensive and diverse evidence. Using a formal qualitative methodology, we interviewed dozens of practitioners from a broad range of backgrounds, all contributors to this novel work of attempting to cause LLMs to fail. We focused on the research questions of defining LLM red teaming, uncovering the motivations and goals for performing the activity, and characterizing the strategies people use when attacking LLMs. Based on the data, LLM red teaming is defined as a limit-seeking, non-malicious, manual activity, which depends highly on a team-effort and an alchemist mindset. It is highly intrinsically motivated by curiosity, fun, and to some degrees by concerns for various harms of deploying LLMs. We identify a taxonomy of 12 strategies and 35 different techniques of attacking LLMs. These findings are presented as a comprehensive grounded theory of how and why people attack large language models: LLM red teaming.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 20 of its 75 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

20 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition24 Oct
  2. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Models19 Oct
  3. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  4. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 16

  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  5. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  7. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  8. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  9. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  10. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  11. 2018Adversarial Examples that Fool both Computer Vision and Time-Limited Humanscited by 73
  12. 2020Conceptual Metaphors Impact Perceptions of Human-AI Collaborationcited by 47
4 more
  1. 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
  2. 2023Assessing Language Model Deployment with Risk Cardscited by 30
  3. 2023The Prompt Artistscited by 21
  4. 2023Designing Participatory AI: Creative Professionals' Worries and Expectations about Generative AIcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 9; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  2. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  3. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  4. 2024garak: A Framework for Security Probing Large Language Modelscited by 48
  5. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  6. 2024When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Cipherscited by 27
  7. 2025OpenAI's Approach to External Red Teaming for AI Models and Systemscited by 26
  8. 2024SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Taskscited by 17
  9. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Modelscited by 16
  10. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  11. 2024Improved Large Language Model Jailbreak Detection via Pretrained Embeddingscited by 10
  12. 2025Red Teaming Contemporary AI Models: Insights from Spanish and Basque Perspectivescited by 6
8 more
  1. 2025Oversight Structures for Agentic AI in Public-Sector Organizationscited by 6
  2. 2024Hacc-Man: An Arcade Game for Jailbreaking LLMscited by 4
  3. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  4. 2025Red Teaming AI Red Teamingcited by 3
  5. 2025NLP Security and Ethics, in the Wildcited by 2
  6. 2025Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Datasetcited by 2
  7. 2025AprielGuardcited by 2
  8. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0