mlsanityData to 7 Oct 2026

Jatmo: Prompt Injection Defense by Task-Specific Finetuning

arXiv 2312.17673 ↗First posted 29 Dec 2023cs.CR cs.AI cs.CLcited by 89

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) are attracting significant research attention due to their instruction-following abilities, allowing users and developers to leverage LLMs for a variety of tasks. However, LLMs are vulnerable to prompt-injection attacks: a class of attacks that hijack the model's instruction-following abilities, changing responses to prompts to undesired, possibly malicious ones. In this work, we introduce Jatmo, a method for generating task-specific models resilient to prompt-injection attacks. Jatmo leverages the fact that LLMs can only follow instructions once they have undergone instruction tuning. It harnesses a teacher instruction-tuned model to generate a task-specific dataset, which is then used to fine-tune a base model (i.e., a non-instruction-tuned model). Jatmo only needs a task prompt and a dataset of inputs for the task: it uses the teacher model to generate outputs. For situations with no pre-existing datasets, Jatmo can use a single example, or in some cases none at all, to produce a fully synthetic dataset. Our experiments on seven tasks show that Jatmo models provide similar quality of outputs on their specific task as standard LLMs, while being resilient to prompt injections. The best attacks succeeded in less than 0.5% of cases against our models, versus 87% success rate against GPT-3.5-Turbo. We release Jatmo at https://github.com/wagner-group/prompt-injection-defense.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 40 of its 57 references that are in the map. First posted on arXiv 29 Dec 2023.

Builds on

40 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  2. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition24 Oct
  3. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defenses19 Oct
  4. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  5. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2015Teaching Machines to Read and Comprehendcited by 1,039
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
22 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  5. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  6. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  7. 2023Detecting Language Model Attacks with Perplexitycited by 411
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  10. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  11. 2023Challenges and Applications of Large Language Modelscited by 251
  12. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  13. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  14. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  15. 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 94
  16. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
  17. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  18. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  19. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  20. 2023Mark My Words: Analyzing and Evaluating Language Model Watermarkscited by 62
  21. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
  22. 2023Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratingscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 40; 2026: 23.

89 papers.

Sort:

By area:

  1. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  2. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  3. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  4. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  5. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  6. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  7. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  8. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  9. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  10. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  11. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  12. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
38 more
  1. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  2. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  4. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  5. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  6. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  7. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  8. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
  9. 2024WIPI: A New Web Threat for LLM-Driven Web Agentscited by 50
  10. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  11. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  12. 2024Boosting Jailbreak Attack with Momentumcited by 38
  13. 2025A Closer Look at System Prompt Robustnesscited by 31
  14. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  15. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  16. 2024Security of AI Agentscited by 24
  17. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
  18. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  19. 2024GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applicationscited by 22
  20. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21
  21. 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 18
  22. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  23. 2025Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworkscited by 14
  24. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  25. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  26. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  27. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  28. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  29. 2025How Not to Detect Prompt Injections with an LLMcited by 11
  30. 2025TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detectioncited by 11
  31. 2025Preventing Prompt Injection with Type-Directed Privilege Separationcited by 8
  32. 2024Applying Pre-trained Multilingual BERT in Embeddings for Improved Malicious Prompt Injection Attacks Detectioncited by 7
  33. 2025Gandalf the Red: Adaptive Security for LLMscited by 7
  34. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMscited by 7
  35. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 7
  36. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systemscited by 7
  37. 2026Poise: Position-Aware One-Instruction Skill Injection for Silent Execution on LLM Agentscited by 7
  38. 2025LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agentscited by 6

50 most cited of 89 All 89 (JSON)