mlsanityData to 7 Oct 2026

PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails

arXiv 2402.15911 ↗First posted 24 Feb 2024cs.CR cs.CLcited by 49

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) are typically aligned to be harmless to humans. Unfortunately, recent work has shown that such models are susceptible to automated jailbreak attacks that induce them to generate harmful content. More recent LLMs often incorporate an additional layer of defense, a Guard Model, which is a second LLM that is designed to check and moderate the output response of the primary LLM. Our key contribution is to show a novel attack strategy, PRP, that is successful against several open-source (e.g., Llama 2) and closed-source (e.g., GPT 3.5) implementations of Guard Models. PRP leverages a two step prefix-based attack that operates by (a) constructing a universal adversarial prefix for the Guard Model, and (b) propagating this prefix to the response. We find that this procedure is effective across multiple threat models, including ones in which the adversary has no access to the Guard Model at all. Our work suggests that further advances are required on defenses and Guard Models before they can be considered effective.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 28 of its 56 references that are in the map. First posted on arXiv 24 Feb 2024.

Builds on

28 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  2. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  5. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  6. 2023Detecting Language Model Attacks with Perplexity27 Aug
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Mistral 7Bcited by 3,065
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
8 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  5. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  6. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  7. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  8. 2019Universal Adversarial Perturbation for Text Classificationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 18; 2026: 4.

49 papers.

Sort:

By area:

  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  4. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  5. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  6. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  7. 2024Safeguarding Large Language Models: A Surveycited by 73
  8. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  9. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  10. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  11. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  12. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
37 more
  1. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  2. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  3. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  4. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  5. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  6. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  7. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 20
  8. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  9. 2024HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Routercited by 16
  10. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  11. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  12. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
  13. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safetycited by 10
  14. 2025Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderationcited by 9
  15. 2025Agents Are All You Need for LLM Unlearningcited by 9
  16. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  17. 2025DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimizationcited by 9
  18. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  19. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 8
  20. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  21. 2024Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approachcited by 6
  22. 2025Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Modelscited by 5
  23. 2024Securing the Future of GenAI: Policy and Technologycited by 4
  24. 2025Mitigating Jailbreaks with Intent-Aware LLMscited by 4
  25. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  26. 2024PolicyLR: A Logic Representation For Privacy Policiescited by 3
  27. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 3
  28. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  29. 2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Awaycited by 3
  30. 2025Beyond the Tip of Efficiency: Uncovering the Submerged Threats of Jailbreak Attacks in Small Language Modelscited by 2
  31. 2026Agent Security is a Systems Problemcited by 2
  32. 2024Output Scouting: Auditing Large Language Models for Catastrophic Responsescited by 1
  33. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  34. 2025Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Modelscited by 0
  35. 2025SoK: a Comprehensive Causality Analysis Framework for Large Language Model Securitycited by 0
  36. 2026Distilling Safe LLM Systems via Soft Prompts for On Device Settingscited by 0
  37. 2026Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detectioncited by 0