mlsanityData to 7 Oct 2026

How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States

arXiv 2406.05644 ↗First posted 9 Jun 2024cs.CL cs.AI cs.CRcited by 92

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large language models (LLMs) rely on safety alignment to avoid responding to malicious user inputs. Unfortunately, jailbreak can circumvent safety guardrails, resulting in LLMs generating harmful content and raising concerns about LLM safety. Due to language models with intensive parameters often regarded as black boxes, the mechanisms of alignment and jailbreak are challenging to elucidate. In this paper, we employ weak classifiers to explain LLM safety through the intermediate hidden states. We first confirm that LLMs learn ethical concepts during pre-training rather than alignment and can identify malicious and normal inputs in the early layers. Alignment actually associates the early concepts with emotion guesses in the middle layers and then refines them to the specific reject tokens for safe generations. Jailbreak disturbs the transformation of early unethical classification into negative emotions. We conduct experiments on models from 7B to 70B across various model families to prove our conclusion. Overall, our paper indicates the intrinsical mechanism of LLM safety and how jailbreaks circumvent safety guardrails, offering a new perspective on LLM safety and reducing concerns. Our code is available at https://github.com/ydyjya/LLM-IHS-Explanation.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 40 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

20 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2023LIMA: Less Is More for Alignmentcited by 1,080
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  12. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
6 more
  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  2. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  3. 2023Explainability for Large Language Models: A Surveycited by 331
  4. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  5. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  6. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 48; 2026: 32.

92 papers.

Sort:

By area:

  1. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  2. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  3. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  4. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  5. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  6. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  7. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  8. 2025Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Taskscited by 18
  9. 2026Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montagecited by 17
  10. 2025Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expressioncited by 16
  11. 2025Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerancecited by 16
  12. 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 15
38 more
  1. 2025Depth Gives a False Sense of Privacy: LLM Internal States Inversioncited by 14
  2. 2024Course-Correction: Safety Alignment Using Synthetic Preferencescited by 13
  3. 2024Quantized Delta Weight Is Safety Keepercited by 12
  4. 2026Valence-Arousal Subspace in LLMs: Circular Emotion Geometry and Multi-Behavioral Controlcited by 12
  5. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  6. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 11
  7. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  8. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  9. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 9
  10. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  11. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 9
  12. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  13. 2024Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuningcited by 8
  14. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 8
  15. 2025Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancementcited by 8
  16. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  17. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 8
  18. 2024LLM-Virus: Evolutionary Jailbreak Attack on Large Language Modelscited by 7
  19. 2025Unraveling LLM Jailbreaks Through Safety Knowledge Neuronscited by 7
  20. 2025Safe at the Margins: A General Approach to Safety Alignment in Low-Resource English Languages -- A Singlish Case Studycited by 6
  21. 2025Adaptive Detoxification: Safeguarding General Capabilities of LLMs through Toxicity-Aware Knowledge Editingcited by 5
  22. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 5
  23. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  24. 2025When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injectioncited by 5
  25. 2026When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agentscited by 5
  26. 2025SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Interventioncited by 4
  27. 2025CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representationscited by 4
  28. 2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4
  29. 2025Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?cited by 4
  30. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Modelscited by 4
  31. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  32. 2025Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learningcited by 3
  33. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Paperscited by 3
  34. 2025Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMscited by 3
  35. 2026ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplificationcited by 3
  36. 2026RASA: Routing-Aware Safety Alignment for Mixture-of-Experts Modelscited by 3
  37. 2026Before the Last Token: Diagnosing Final-Token Safety Probe Failurescited by 3
  38. 2024Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributionscited by 2

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Circuit Discovery Helps Detect LLM Jailbreaking: A Mechanistic Interpretability Study
  2. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?
  3. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
  4. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  5. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
  6. 2025TwinBreak: Jailbreaking LLM Security Alignments based on Twin Prompts
  7. 2026Exploring and Developing a Pre-Model Safeguard with Draft Models
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep