mlsanityData to 7 Oct 2026

Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak

arXiv 2607.14147 ↗First posted 14 Jul 2026cs.CL cs.AI cs.CRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Aligned language models refuse harmful requests, but a one-line prefill ("Sure, here is") strips the refusal. We ask where and how it fails. The harm representation stays intact: on the prompts the attack flips to compliance, a linear probe reads harm as high as on the refused ones (0.91-0.98), while behavioral refusal drops to chance. This holds across four models and three families (1.5-3.8B, and at 14B). Refusal is therefore a shallow, response-site computation. We localize it to an early window: a dose-matched position control shows the first half of the response suffices to break refusal, while the second half is nearly inert. Three causal probes converge on that window. Restoring the harm direction there partially re-engages refusal. Injecting the model's own refuse-state reverses the jailbreak (74%, held-out). And knocking out the early response's attention to the prefill, but not an equal attention mass elsewhere, selectively collapses the harmful continuation. A base-model control identifies the mechanism: the same knockout collapses the continuation prefill-specifically even in a non-safety-tuned base model (64% to 25% harmful content vs a matched control's 64%, replicated at 7B). So the prefill's grip is generic autoregressive conditioning, not safety-specific suppression, and "refusal restoration" is a model-dependent fallback. The dominant mechanism is passive. A small safety-specific attractor remains on top (logit-trace concentration 0.24 vs 0.03), whose active-vs-passive character we size but do not fully separate. No single direction or component is a clean handle either: the decision is decodable but distributed, and refusal tracks harm rather than scary surface. The consequence is structural: a monitor reading the untouched prompt-side representation is immune by construction, but only to response-site attacks. The mechanism is diffuse; the failure surface is local.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 30 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

25 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth20 Oct
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 22

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  11. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  12. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
10 more
  1. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulationcited by 46
  2. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  3. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  4. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  5. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 24
  6. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
  7. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Modelscited by 9
  8. 2025Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attackscited by 5
  9. 2026Before the Last Token: Diagnosing Final-Token Safety Probe Failurescited by 3
  10. 2026They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On Itcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off
  2. 2026Latent-space Attacks for Refusal Evasion in Language Models
  3. 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks
  4. 2026Tool Mediation Alters Refusal Mechanisms in Large Language Models
  5. 2025Refusal Direction is Universal Across Safety-Aligned Languages
  6. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
  7. 2026Refusal Localizes, the Damage Relocates: Safety Layers Under Few-Sample Fine-Tuning
  8. 2026Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs