mlsanityData to 7 Oct 2026

Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

arXiv 2606.04778 ↗First posted 3 Jun 2026cs.AI cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-token generation. Our work argues that robust safety alignment requires training on the generation process itself, not only its outputs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 36 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

30 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMs19 Feb
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
14 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  5. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  6. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  7. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  8. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  9. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  10. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  11. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  12. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  13. 2025CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representationscited by 4
  14. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Autoregressive Consistency Hurts Safety Alignment
  2. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Region
  3. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing
  4. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
  5. 2025Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
  6. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignment
  7. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time
  8. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspective