mlsanityData to 14 Oct 2026

Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis

arXiv 2406.10794 ↗First posted 16 Jun 2024cs.CLcited by 64

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. Although there are diverse jailbreak attack strategies, there is no unified understanding on why some methods succeed and others fail. This paper explores the behavior of harmful and harmless prompts in the LLM's representation space to investigate the intrinsic properties of successful jailbreak attacks. We hypothesize that successful attacks share some similar properties: They are effective in moving the representation of the harmful prompt towards the direction to the harmless prompts. We leverage hidden representations into the objective of existing jailbreak attacks to move the attacks along the acceptance direction, and conduct experiments to validate the above hypothesis using the proposed objective. We hope this study provides new insights into understanding how LLMs understand harmfulness information.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 20 of its 36 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

20 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  12. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
4 more
  1. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  2. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  3. 2023Pretraining Language Models with Human Preferencescited by 247
  4. 2023Using Large Language Models to Support Thematic Analysis in Empirical Legal Studiescited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 31; 2026: 24.

64 papers.

Sort:

By area:

  1. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulationcited by 54
  2. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  3. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 43
  4. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 33
  5. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  6. 2025Reasoning as an Adaptive Defense for Safetycited by 23
  7. 2024JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuitcited by 22
  8. 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspectivecited by 20
  9. 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 19
  10. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 18
  11. 2025Chain-of-Thought Hijackingcited by 16
  12. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 15
38 more
  1. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 13
  2. 2025Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Trainingcited by 12
  3. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  4. 2024Towards Understanding the Robustness of LLM-based Evaluations under Perturbationscited by 10
  5. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  6. 2025Unraveling LLM Jailbreaks Through Safety Knowledge Neuronscited by 7
  7. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  8. 2025Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approachcited by 6
  9. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual contextcited by 6
  10. 2024Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMscited by 5
  11. 2025CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representationscited by 5
  12. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injectioncited by 5
  13. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  14. 2025Characterizing Selective Refusal Bias in Large Language Modelscited by 4
  15. 2025Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimizationcited by 4
  16. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  17. 2025Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralizationcited by 3
  18. 2026Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defensecited by 3
  19. 2026AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agentcited by 3
  20. 2025Linearly Decoding Refused Knowledge in Aligned Language Modelscited by 2
  21. 2025Dual-Space Smoothness for Robust and Balanced LLM Unlearningcited by 2
  22. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignmentcited by 2
  23. 2026Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokenscited by 2
  24. 2026Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometrycited by 2
  25. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  26. 2024Defending against Jailbreak through Early Exit Generation of Large Language Modelscited by 1
  27. 2025NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Modelscited by 1
  28. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  29. 2025Bolster Hallucination Detection via Prompt-Guided Data Augmentationcited by 1
  30. 2025Toward Understanding the Transferability of Adversarial Suffixes in Large Language Modelscited by 1
  31. 2025One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMscited by 1
  32. 2026SafeThinker: Reasoning about Risk to Deepen Safety Beyond Shallow Alignmentcited by 1
  33. 2026ShallowJail: Steering Jailbreaks against Large Language Modelscited by 1
  34. 2026Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correctioncited by 1
  35. 2024Transferable & Stealthy Ensemble Attacks: A Black-Box Jailbreaking Framework for Large Language Modelscited by 0
  36. 2025TombRaider: Entering the Vault of History to Jailbreak Large Language Modelscited by 0
  37. 2025SoK: a Comprehensive Causality Analysis Framework for Large Language Model Securitycited by 0
  38. 2026ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generationcited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective
  2. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions
  3. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks
  4. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs
  5. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
  6. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
  7. 2024An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks
  8. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures