mlsanityData to 7 Oct 2026

Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game

arXiv 2311.01011 ↗First posted 2 Nov 2023cs.LG cs.CRcited by 135

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

While Large Language Models (LLMs) are increasingly being used in real-world applications, they remain vulnerable to prompt injection attacks: malicious third party prompts that subvert the intent of the system designer. To help researchers study this problem, we present a dataset of over 126,000 prompt injection attacks and 46,000 prompt-based "defenses" against prompt injection, all created by players of an online game called Tensor Trust. To the best of our knowledge, this is currently the largest dataset of human-generated adversarial examples for instruction-following LLMs. The attacks in our dataset have a lot of easily interpretable stucture, and shed light on the weaknesses of LLMs. We also use the dataset to create a benchmark for resistance to two types of prompt injection, which we refer to as prompt extraction and prompt hijacking. Our benchmark results show that many models are vulnerable to the attack strategies in the Tensor Trust dataset. Furthermore, we show that some attack strategies from the dataset generalize to deployed LLM-based applications, even though they have a very different set of constraints to the game. We release all data and source code at https://tensortrust.ai/paper

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 16 of its 41 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

16 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  10. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  12. 2019A backdoor attack against LSTM-based text classification systemscited by 214
2 more
  1. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  2. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 50; 2025: 26; 2026: 57.

135 papers.

Sort:

By area:

  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  2. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  4. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  5. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  6. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  7. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  8. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  9. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  10. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  11. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  12. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
38 more
  1. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  2. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  3. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  4. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  5. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  6. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  7. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  8. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  9. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  10. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  11. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 60
  12. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  13. 2024WIPI: A New Web Threat for LLM-Driven Web Agentscited by 50
  14. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  15. 2023Can LLMs Follow Simple Rules?cited by 45
  16. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  17. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
  18. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  19. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  20. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  21. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  22. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 33
  23. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 33
  24. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  25. 2025A Closer Look at System Prompt Robustnesscited by 31
  26. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 31
  27. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  28. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purificationcited by 28
  29. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 26
  30. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  31. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  32. 2024Security of AI Agentscited by 24
  33. 2024Ranking Manipulation for Conversational Search Enginescited by 23
  34. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 23
  35. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  36. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  37. 2024GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applicationscited by 22
  38. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21

50 most cited of 135 All 135 (JSON)