mlsanityData to 14 Oct 2026

A Closer Look at System Prompt Robustness

arXiv 2502.12197 ↗First posted 15 Feb 2025cs.CL cs.AIcited by 32

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

System prompts have emerged as a critical control surface for specifying the behavior of LLMs in chat and agent settings. Developers depend on system prompts to specify important context, output format, personalities, guardrails, content policies, and safety countermeasures, all of which require models to robustly adhere to the system prompt, especially when facing conflicting or adversarial user inputs. In practice, models often forget to consider relevant guardrails or fail to resolve conflicting demands between the system and the user. In this work, we study various methods for improving system prompt robustness by creating realistic new evaluation and fine-tuning datasets based on prompts collected from from OpenAI's GPT Store and HuggingFace's HuggingChat. Our experiments assessing models with a panel of new and existing benchmarks show that performance can be considerably improved with realistic fine-tuning data, as well as inference-time interventions such as classifier-free guidance. Finally, we analyze the results of recently released reasoning models from OpenAI and DeepSeek, which show exciting but uneven improvements on the benchmarks we study. Overall, current techniques fall short of ensuring system prompt robustness and further study is warranted.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 44 of its 50 references that are in the map. First posted on arXiv 15 Feb 2025.

Builds on

44 of 50 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy9 Oct
  2. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  3. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  5. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning29 Dec
  6. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  7. 2023Assessing Prompt Injection Risks in 200+ Custom GPTs20 Nov
  8. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  9. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection17 Aug
  10. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  11. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
21 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  3. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  4. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  5. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  6. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  11. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  12. 2023Stay on topic with Classifier-Free Guidancecited by 91
  13. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  14. 2020Learning from Task Descriptionscited by 78
  15. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69
  16. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 61
  17. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 53
  18. 2023Can LLMs Follow Simple Rules?cited by 46
  19. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 29
  20. 2024Is the System Message Really Important to Jailbreaks in Large Language Models?cited by 20
  21. 2024SoFA: Shielded On-the-fly Alignment via Priority Rule Followingcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 19.

32 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  2. 2025Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMscited by 22
  3. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
  4. 2025The Pitfalls of KV Cache Compressioncited by 15
  5. 2026IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMscited by 15
  6. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attackscited by 10
  7. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Pluginscited by 9
  8. 2026Detecting Malicious Agent Skills in the Wild using Attentioncited by 8
  9. 2026Multi-User Large Language Model Agentscited by 7
  10. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  11. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  12. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3
20 more
  1. 2025Combining TSL and LLM to Automate REST API Testing: A Comparative Studycited by 3
  2. 2026HIPO: Instruction Hierarchy via Constrained Reinforcement Learningcited by 3
  3. 2026Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Taskscited by 3
  4. 2026SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillationcited by 3
  5. 2025Research Challenges in Relational Database Management Systems for LLM Queriescited by 2
  6. 2026Position: Capability Control Should be a Separate Goal From Alignmentcited by 2
  7. 2026Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defensecited by 2
  8. 2026Steering Instruction Hierarchies at Inference Timecited by 2
  9. 2026Context Cartography: Toward Structured Governance of Contextual Space in Large Language Model Systemscited by 1
  10. 2026Bias in the Tails: How Name-conditioned Evaluative Framing in Resume Summaries Destabilizes LLM-based Hiringcited by 1
  11. 2026RogueMerge: Robust and Unified Attacks against LLM Model Mergingcited by 1
  12. 2025Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptorcited by 0
  13. 2025Highlight & Summarize: RAG without the jailbreakscited by 0
  14. 2025When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agentscited by 0
  15. 2026Rewriting the Response Path: Silent Tampering and Provider-Signed Defense in BYOK LLM Agentscited by 0
  16. 2026Training ML Models with Predictable Failurescited by 0
  17. 2026WIRE: Profiling Witnessed Within-Policy Instruction Collisions in LLM Agentscited by 0
  18. 2026Where Instruction Hierarchy Breaks: Diagnosing and Repairing Failures in Reasoning Language Modelscited by 0
  19. 2026SyRuP: Enhancing System-Prompt Following via Reward-Guided Prediction in LLM Decodingcited by 0
  20. 2026AgentPersonaBench: Benchmarking Persona-Driven User Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks
  2. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
  3. 2025Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
  4. 2026Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity
  5. 2026Same Question, Different Answer? Measuring and Mitigating Prompt Privilege for Equitable AI Access
  6. 2025The Meta-Prompting Protocol: Orchestrating LLMs via Adversarial Feedback Loops
  7. 2025Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Prompts
  8. 2025Breaking the Prompt Wall (I): A Real-World Case Study of Attacking ChatGPT via Lightweight Prompt Injection