Data to 14 Oct 2026

Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection

arXiv 2308.10819 ↗First posted 17 Aug 2023cs.CL cs.AIcited by 59

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, becoming increasingly crucial across various applications. However, this capability brings with it the risk of prompt injection attacks, where attackers inject instructions into LLMs' input to elicit undesirable actions or content. Understanding the robustness of LLMs against such attacks is vital for their safe implementation. In this work, we establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks. Our objective is to determine the extent to which LLMs can be influenced by injected instructions and their ability to differentiate between these injected and original target instructions. Through extensive experiments with leading instruction-following LLMs, we uncover significant vulnerabilities in their robustness to such attacks. Our results indicate that some models are overly tuned to follow any embedded instructions in the prompt, overly focusing on the latter parts of the prompt without fully grasping the entire context. By contrast, models with a better grasp of the context and instruction-following capabilities will potentially be more susceptible to compromise by injected instructions. This underscores the need to shift the focus from merely enhancing LLMs' instruction-following capabilities to improving their overall comprehension of prompts and discernment of instructions that are appropriate to follow. We hope our in-depth analysis offers insights into the underlying causes of these vulnerabilities, aiding in the development of future solutions. Code and data are available at https://github.com/Leezekun/instruction-following-robustness-eval

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 28 of its 40 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

28 of 40 references.

Closest ancestors · 1

Same direction, newest first

2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr

Other references · 27

2023GPT-4 Technical Reportcited by 18,396
2022Training language models to follow instructions with human feedbackcited by 15,971
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
2021Improving language models by retrieving from trillions of tokenscited by 1,293

15 more

2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
2023Instruction Tuning with GPT-4cited by 686
2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
2022News Summarization and Evaluation in the Era of GPT-3cited by 327
2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
2023On the Exploitability of Instruction Tuningcited by 116
2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 66
2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 25; 2026: 21.

59 papers.

Sort:

By area:

2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
2025Unveiling Privacy Risks in LLM Agent Memorycited by 101
2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 55
2025Can Indirect Prompt Injection Attacks Be Detected and Removed?cited by 52
2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
2024Defense Against Prompt Injection Attack by Leveraging Attack Techniquescited by 40
2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
2025A Closer Look at System Prompt Robustnesscited by 32
2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 32

38 more

2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 30
2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 21
2025TopicAttack: An Indirect Prompt Injection Attack via Topic Transitioncited by 17
2024Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applicationscited by 16
2025Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instructioncited by 16
2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
2025Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendationscited by 11
2026HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?cited by 11
2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?cited by 10
2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
2025PRL: Prompts from Reinforcement Learningcited by 9
2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
2024UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughtscited by 8
2025DLPO: Towards a Robust, Efficient, and Generalizable Prompt Optimization Framework from a Deep-Learning Perspectivecited by 8
2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
2025EduGuardBench: A Holistic Benchmark for Evaluating the Pedagogical Fidelity and Adversarial Safety of LLMs as Simulated Teacherscited by 5
2025Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformationcited by 4
2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
2026Neuro-Symbolic Verification on Instruction Following of LLMscited by 4
2026JevOut: Natural Context Can Flip Decision Modelscited by 4
2025Backdoor-Powered Prompt Injection Attacks Nullify Defense Methodscited by 3
2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
2024LUK: Empowering Log Understanding with Expert Knowledge from Large Language Modelscited by 2
2025How does Watermarking Affect Visual Language Models in Document Understanding?cited by 2
2025Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separatorscited by 2
2025When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMscited by 2
2025The Trust in AI-Generated Health Advice (TAIGHA) Scale and Short Version (TAIGHA-S): Development and Validation Studycited by 1
2026TATRA: Training-Free Instance-Adaptive Prompting Through Rephrasing and Aggregationcited by 1
2026Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Headscited by 1
2026PLR: Plackett-Luce for Reordering In-Context Learning Examplescited by 1
2026PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 1
2026"**Important** You should give me full credits!": Exploring Prompt Injection Attacks on LLM-Based Automatic Grading Systemscited by 1
2026IACM-RL: Intent-Aware Context Management and Reinforcement Learning for Complex Tool Invocation under Dynamic Intent Fluctuationscited by 1
2026LongPIBench: A Long-Context Benchmark for Prompt Injectioncited by 1
2025Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMscited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way