mlsanityData to 14 Oct 2026

Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition

arXiv 2406.07954 ↗First posted 12 Jun 2024cs.CR cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Large language model systems face important security risks from maliciously crafted messages that aim to overwrite the system's original instructions or leak private data. To study this problem, we organized a capture-the-flag competition at IEEE SaTML 2024, where the flag is a secret string in the LLM system prompt. The competition was organized in two phases. In the first phase, teams developed defenses to prevent the model from leaking the secret. During the second phase, teams were challenged to extract the secrets hidden for defenses proposed by the other teams. This report summarizes the main insights from the competition. Notably, we found that all defenses were bypassed at least once, highlighting the difficulty of designing a successful defense and the necessity for additional research to protect LLM systems. To foster future research in this direction, we compiled a dataset with over 137k multi-turn attack chats and open-sourced the platform.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

21 of 31 references.

References · 21

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  4. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  5. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  6. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  8. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  9. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  10. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  11. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  12. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
9 more
  1. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171
  2. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 145
  3. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 81
  4. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 72
  5. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69
  6. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  7. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 49
  8. 2023Privacy Side Channels in Machine Learning Systemscited by 40
  9. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025Defeating Prompt Injections by Designcited by 297
  2. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  4. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  5. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 90
  6. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 63
  7. 2024Persistent Pre-Training Poisoning of LLMscited by 48
  8. 2025Firewalls to Secure Dynamic LLM Agentic Networkscited by 36
  9. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 26
  10. 2025LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challengecited by 21
  11. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  12. 2024Prompt Obfuscation for Large Language Modelscited by 10
8 more
  1. 2024Applying Refusal-Vector Ablation to Llama 3.1 70B Agentscited by 6
  2. 2024Hey GPT, Can You be More Racist? Analysis from Crowdsourced Attempts to Elicit Biased Content from Generative AIcited by 6
  3. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  4. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
  5. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltrationcited by 3
  6. 2026Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformationscited by 3
  7. 2026Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challengecited by 2
  8. 2026No More, No Less: Task Alignment in Terminal Agentscited by 2