mlsanityData to 14 Oct 2026

On Prompt-Driven Safeguarding for Large Language Models

arXiv 2401.18018 ↗First posted 31 Jan 2024cs.LG cs.AI cs.CLcited by 116

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Prepending model inputs with safety prompts is a common practice for safeguarding large language models (LLMs) against queries with harmful intents. However, the underlying working mechanisms of safety prompts have not been unraveled yet, restricting the possibility of automatically optimizing them to improve LLM safety. In this work, we investigate how LLMs' behavior (i.e., complying with or refusing user queries) is affected by safety prompts from the perspective of model representation. We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction, in which models become more prone to refusing to provide assistance, even when the queries are harmless. On the other hand, LLMs are naturally capable of distinguishing harmful and harmless queries without safety prompts. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO (Directed Representation Optimization). Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness. Experiments with eight LLMs on out-of-domain and jailbreak benchmarks demonstrate that DRO remarkably improves the safeguarding performance of human-crafted safety prompts, without compromising the models' general performance.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 33 of its 38 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

33 of 38 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Mistral 7Bcited by 3,164
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
21 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 966
  2. 2023Large Language Models as Optimizerscited by 864
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  8. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  9. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  11. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  12. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  13. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  14. 2023Self-Alignment with Instruction Backtranslationcited by 151
  15. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  16. 2020Towards Controllable Biases in Language Generationcited by 112
  17. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  18. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  19. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  20. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  21. 2021Exploring Prompt-based Few-shot Learning for Grounded Dialog Generationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 42; 2025: 50; 2026: 24.

116 papers.

Sort:

By area:

  1. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  2. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  4. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  5. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  6. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  7. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  8. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  9. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  10. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  11. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 67
  12. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
38 more
  1. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  2. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  3. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 58
  4. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulationcited by 54
  5. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  6. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  7. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistencycited by 51
  8. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  9. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 47
  10. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 46
  11. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 46
  12. 2025LLM Unlearning via Neural Activation Redirectioncited by 41
  13. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 37
  14. 2024Personality Alignment of Large Language Modelscited by 36
  15. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 34
  16. 2025MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Serverscited by 34
  17. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 33
  18. 2025Defending Against Prompt Injection With a Few DefensiveTokenscited by 33
  19. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 33
  20. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 31
  21. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 29
  22. 2024From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attackscited by 29
  23. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 29
  24. 2024Perturbation-Restrained Sequential Model Editingcited by 28
  25. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attackscited by 28
  26. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 26
  27. 2025Steering MoE LLMs via Expert (De)Activationcited by 26
  28. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Modelscited by 23
  29. 2024Locking Down the Finetuned LLMs Safetycited by 22
  30. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  31. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 22
  32. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 18
  33. 2025PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Modelscited by 17
  34. 2025Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerancecited by 17
  35. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 16
  36. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 15
  37. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
  38. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 14

50 most cited of 116 All 116 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mitigating Exaggerated Safety in Large Language Models
  2. 2025Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Prompts
  3. 2026Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMs
  4. 2026Enhancing Safety of Large Language Models via Embedding Space Separation
  5. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  6. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  7. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
  8. 2026Safety Targeted Embedding Exploit via Refinement