mlsanityData to 14 Oct 2026

Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield

arXiv 2311.00172 ↗First posted 31 Oct 2023cs.CL cs.AIcited by 25

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large Language Models' safety remains a critical concern due to their vulnerability to adversarial attacks, which can prompt these systems to produce harmful responses. In the heart of these systems lies a safety classifier, a computational model trained to discern and mitigate potentially harmful, offensive, or unethical outputs. However, contemporary safety classifiers, despite their potential, often fail when exposed to inputs infused with adversarial noise. In response, our study introduces the Adversarial Prompt Shield (APS), a lightweight model that excels in detection accuracy and demonstrates resilience against adversarial prompts. Additionally, we propose novel strategies for autonomously generating adversarial training datasets, named Bot Adversarial Noisy Dialogue (BAND) datasets. These datasets are designed to fortify the safety classifier's robustness, and we investigate the consequences of incorporating adversarial examples into the training process. Through evaluations involving Large Language Models, we demonstrate that our classifier has the potential to decrease the attack success rate resulting from adversarial attacks by up to 60%. This advancement paves the way for the next generation of more reliable and resilient conversational agents.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 15 of its 24 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

15 of 24 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  12. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
3 more
  1. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  2. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  3. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 7; 2026: 4.

25 papers.

Sort:

By area:

  1. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  2. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  4. 2024Building Guardrails for Large Language Modelscited by 65
  5. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  6. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 44
  7. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
  8. 2024Legilimens: Practical and Unified Content Moderation for Large Language Model Servicescited by 13
  9. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  10. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 9
  11. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  12. 2024LocalRQA: From Generating Data to Locally Training, Testing, and Deploying Retrieval-Augmented QA Systemscited by 7
13 more
  1. 2024GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attackscited by 7
  2. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  3. 2024An Active Learning Framework for Inclusive Generation by Large Language Modelscited by 5
  4. 2025Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Modelscited by 5
  5. 2025JavelinGuard: Low-Cost Transformer Architectures for LLM Securitycited by 4
  6. 2025Prompt Inject Detection with Generative Explanation as an Investigative Toolcited by 2
  7. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  8. 2024CFSafety: Comprehensive Fine-grained Safety Assessment for LLMscited by 1
  9. 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 1
  10. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  11. 2024Doppelg"anger's Watch: A Split Objective Approach to Large Language Modelscited by 0
  12. 2024DIESEL -- Dynamic Inference-Guidance via Evasion of Semantic Embeddings in LLMscited by 0
  13. 2026LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SGuard-v1: Safety Guardrail for Large Language Models
  2. 2025SHIELD: Classifier-Guided Prompting for Robust and Safer LVLMs
  3. 2026Evaluating Language Model Safety Across Long Adversarial Conversations
  4. 2026When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models
  5. 2026Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety
  6. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signals
  7. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models
  8. 2024Robustness of Large Language Models Against Adversarial Attacks