mlsanityData to 7 Oct 2026

SecAlign: Defending Against Prompt Injection with Preference Optimization

arXiv 2410.05451 ↗First posted 7 Oct 2024cs.CR cs.LGcited by 179

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large language models (LLMs) are becoming increasingly prevalent in modern software systems, interfacing between the user and the Internet to assist with tasks that require advanced language understanding. To accomplish these tasks, the LLM often uses external data sources such as user documents, web retrieval, results from API calls, etc. This opens up new avenues for attackers to manipulate the LLM via prompt injection. Adversarial prompts can be injected into external data sources to override the system's intended instruction and instead execute a malicious instruction. To mitigate this vulnerability, we propose a new defense called SecAlign based on the technique of preference optimization. Our defense first constructs a preference dataset with prompt-injected inputs, secure outputs (ones that respond to the legitimate instruction), and insecure outputs (ones that respond to the injection). We then perform preference optimization on this dataset to teach the LLM to prefer the secure output over the insecure one. This provides the first known method that reduces the success rates of various prompt injections to <10%, even against attacks much more sophisticated than ones seen during training. This indicates our defense generalizes well against unknown and yet-to-come attacks. Also, SecAlign models are still practical with similar utility to the one before defensive training in our evaluations. Our code is at https://github.com/facebookresearch/SecAlign

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 82 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

49 of 82 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
34 more
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  7. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  8. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  9. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  10. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  11. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  12. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  13. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  14. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  15. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  16. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  17. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  18. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  19. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  20. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  21. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  22. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  23. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  24. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  25. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  26. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  27. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
  28. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  29. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  30. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  31. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  32. 2023Assessing Prompt Injection Risks in 200+ Custom GPTscited by 49
  33. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  34. 2024A Novel Evaluation Framework for Assessing Resilience Against Prompt Injection Attacks in Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 49; 2026: 129.

179 papers.

Sort:

By area:

  1. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  2. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  3. 2025Securing AI Agents with Information-Flow Controlcited by 108
  4. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  6. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 59
  7. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  8. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
  9. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  10. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  11. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  12. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
38 more
  1. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  2. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31
  3. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  4. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agentscited by 28
  5. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  6. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  7. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  8. 2026Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threatscited by 25
  9. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
  10. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21
  11. 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 18
  12. 2025Enhancing LLM Robustness to Perturbed Instructions: An Empirical Studycited by 17
  13. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 17
  14. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 17
  15. 2026MalTool: Malicious Tool Attacks on LLM Agentscited by 17
  16. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  17. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  18. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  19. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  20. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomycited by 14
  21. 2026Ghost in the Agent: Redefining Information Flow Tracking for LLM Agentscited by 14
  22. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  23. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 13
  24. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  25. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 12
  26. 2025CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitizationcited by 12
  27. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  28. 2026Trojan Hippo: Weaponizing Agent Memory for Data Exfiltrationcited by 12
  29. 2026Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agentscited by 12
  30. 2025TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detectioncited by 11
  31. 2026Prompt Injection as Role Confusioncited by 11
  32. 2026Defenses Against Prompt Attacks Learn Surface Heuristicscited by 10
  33. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  34. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  35. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 10
  36. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysiscited by 9
  37. 2025Systems Security Foundations for Agentic Computingcited by 9
  38. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attackscited by 9

50 most cited of 179 All 179 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LocalAlign: Enabling Generalizable Prompt Injection Defense via Generation of Near-Target Adversarial Examples for Alignment Training
  2. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment
  3. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
  4. 2026Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning
  5. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
  6. 2025A methodological analysis of prompt perturbations and their effect on attack success rates
  7. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
  8. 2025SecInfer: Preventing Prompt Injection via Inference-time Scaling