mlsanityData to 7 Oct 2026

Bypassing the Safety Training of Open-Source LLMs with Priming Attacks

arXiv 2312.12321 ↗First posted 19 Dec 2023cs.CR cs.AI cs.CLcited by 48

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

With the recent surge in popularity of LLMs has come an ever-increasing need for LLM safety training. In this paper, we investigate the fragility of SOTA open-source LLMs under simple, optimization-free attacks we refer to as textitpriming attacks, which are easy to execute and effectively bypass alignment from safety training. Our proposed attack improves the Attack Success Rate on Harmful Behaviors, as measured by Llama Guard, by up to 3.3× compared to baselines. Source code and data are available at https://github.com/uiuc-focal-lab/llm-priming-attacks.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 14 of its 22 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

14 of 22 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 10

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  8. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  9. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?cited by 25
  10. 2023Alignment is not sufficient to prevent large language models from generating harmful information: A psychoanalytic perspectivecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 23; 2026: 13.

48 papers.

Sort:

By area:

  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  4. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  5. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  6. 2024Building Guardrails for Large Language Modelscited by 61
  7. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  8. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  9. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  10. 2024Backtracking Improves Generation Safetycited by 31
  11. 2025Enforcing Temporal Constraints for LLM Agentscited by 31
  12. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templatescited by 29
36 more
  1. 2025Representation Bending for Large Language Model Safetycited by 28
  2. 2025One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMscited by 27
  3. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  4. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injectioncited by 23
  5. 2024HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Routercited by 16
  6. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 13
  7. 2025Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoningcited by 11
  8. 2025Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Modelscited by 10
  9. 2024Certifying Counterfactual Bias in LLMscited by 9
  10. 2024Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Modelscited by 9
  11. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Modelscited by 9
  12. 2025A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Modelscited by 9
  13. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 9
  14. 2025Discovering Forbidden Topics in Language Modelscited by 8
  15. 2026What Matters For Safety Alignment?cited by 6
  16. 2026Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillationcited by 6
  17. 2024Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignmentcited by 5
  18. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoningcited by 4
  19. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuningcited by 4
  20. 2025Fast Proxies for LLM Robustness Evaluationcited by 3
  21. 2026MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMscited by 2
  22. 2024Certifying Knowledge Comprehension in LLMscited by 1
  23. 2025A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokenscited by 1
  24. 2025Saffron-1: Safety Inference Scalingcited by 1
  25. 2025Lumos: Let there be Language Model System Certificationcited by 1
  26. 2026ShallowJail: Steering Jailbreaks against Large Language Modelscited by 1
  27. 2026DRIV-EX: Counterfactual Explanations for Driving LLMscited by 1
  28. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1
  29. 2026Internal-State Probes Read the Situation, Not the Action: Three Negative Results for Pre-Action Misalignment Monitoringcited by 1
  30. 2025Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Modelscited by 0
  31. 2025Matching Ranks Over Probability Yields Truly Deep Safety Alignmentcited by 0
  32. 2026Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimizationcited by 0
  33. 2026Tracing the Dynamics of Refusal: Exploiting Latent Refusal Trajectories for Robust Jailbreak Detectioncited by 0
  34. 2026Few-Shot Truly Benign DPO Attack for Jailbreaking LLMscited by 0
  35. 2026Wait, am I Being Fair? Characterizing Deductive Stereotyping and Mitigating It with Fair-GCGcited by 0
  36. 2026A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformationscited by 0