# Bypassing the Safety Training of Open-Source LLMs with Priming Attacks

arXiv: https://arxiv.org/abs/2312.12321 · HTML: https://mlsanity.com/paper/2312.12321/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 14 of 14)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [MART: Improving LLM Safety with Multi-round Automatic Red-Teaming](https://mlsanity.com/paper/2311.07689/) (2023)
- [On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?](https://mlsanity.com/paper/2310.01581/) (2023)
- [Alignment is not sufficient to prevent large language models from generating harmful information: A psychoanalytic perspective](https://mlsanity.com/paper/2311.08487/) (2023)

## Built on by (most cited first, 30 of 50)

- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [SafeArena: Evaluating the Safety of Autonomous Web Agents](https://mlsanity.com/paper/2503.04957/) (2025)
- [The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness](https://mlsanity.com/paper/2401.00287/) (2023)
- [Building Guardrails for Large Language Models](https://mlsanity.com/paper/2402.01822/) (2024)
- [FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning](https://mlsanity.com/paper/2505.08054/) (2025)
- [Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety](https://mlsanity.com/paper/2503.05021/) (2025)
- [The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMs](https://mlsanity.com/paper/2507.11097/) (2025)
- [Enforcing Temporal Constraints for LLM Agents](https://mlsanity.com/paper/2512.23738/) (2025)
- [Backtracking Improves Generation Safety](https://mlsanity.com/paper/2409.14586/) (2024)
- [ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates](https://mlsanity.com/paper/2406.12935/) (2024)
- [Representation Bending for Large Language Model Safety](https://mlsanity.com/paper/2504.01550/) (2025)
- [One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMs](https://mlsanity.com/paper/2502.18862/) (2025)
- [Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection](https://mlsanity.com/paper/2507.02844/) (2025)
- [SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment](https://mlsanity.com/paper/2505.14667/) (2025)
- [HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router](https://mlsanity.com/paper/2410.02684/) (2024)
- [Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives](https://mlsanity.com/paper/2511.06626/) (2025)
- [Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training](https://mlsanity.com/paper/2510.20956/) (2025)
- [Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoning](https://mlsanity.com/paper/2505.17406/) (2025)
- [Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation](https://mlsanity.com/paper/2605.15239/) (2026)
- [Certifying Counterfactual Bias in LLMs](https://mlsanity.com/paper/2405.18780/) (2024)
- [Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models](https://mlsanity.com/paper/2411.11496/) (2024)
- [Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models](https://mlsanity.com/paper/2504.21038/) (2025)
- [Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models](https://mlsanity.com/paper/2507.05248/) (2025)
- [Discovering Forbidden Topics in Language Models](https://mlsanity.com/paper/2505.17441/) (2025)
- [A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models](https://mlsanity.com/paper/2509.23286/) (2025)
- [Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction](https://mlsanity.com/paper/2509.15202/) (2025)
- [What Matters For Safety Alignment?](https://mlsanity.com/paper/2601.03868/) (2026)
- [Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment](https://mlsanity.com/paper/2411.02785/) (2024)

## Abstract (arXiv)

With the recent surge in popularity of LLMs has come an ever-increasing need for LLM safety training. In this paper, we investigate the fragility of SOTA open-source LLMs under simple, optimization-free attacks we refer to as textitpriming attacks, which are easy to execute and effectively bypass alignment from safety training. Our proposed attack improves the Attack Success Rate on Harmful Behaviors, as measured by Llama Guard, by up to 3.3× compared to baselines. Source code and data are available at https://github.com/uiuc-focal-lab/llm-priming-attacks.