# Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis

arXiv: https://arxiv.org/abs/2406.10794 · HTML: https://mlsanity.com/paper/2406.10794/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies](https://mlsanity.com/paper/2310.18729/) (2023)

## Built on by (most cited first, 30 of 64)

- [JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation](https://mlsanity.com/paper/2502.07557/) (2025)
- [AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint](https://mlsanity.com/paper/2506.07022/) (2025)
- [BadRobot: Jailbreaking Embodied LLM Agents in the Physical World](https://mlsanity.com/paper/2407.20242/) (2024)
- [Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models](https://mlsanity.com/paper/2412.05934/) (2024)
- [Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models](https://mlsanity.com/paper/2412.17034/) (2024)
- [Reasoning as an Adaptive Defense for Safety](https://mlsanity.com/paper/2507.00971/) (2025)
- [JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuit](https://mlsanity.com/paper/2411.11114/) (2024)
- [Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective](https://mlsanity.com/paper/2411.14572/) (2024)
- [False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize](https://mlsanity.com/paper/2509.03888/) (2025)
- [Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary](https://mlsanity.com/paper/2505.18325/) (2025)
- [Chain-of-Thought Hijacking](https://mlsanity.com/paper/2510.26418/) (2025)
- [AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning](https://mlsanity.com/paper/2507.14987/) (2025)
- [An Embarrassingly Simple Defense Against LLM Abliteration Attacks](https://mlsanity.com/paper/2505.19056/) (2025)
- [Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training](https://mlsanity.com/paper/2502.11455/) (2025)
- [Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring](https://mlsanity.com/paper/2512.12069/) (2025)
- [Towards Understanding the Robustness of LLM-based Evaluations under Perturbations](https://mlsanity.com/paper/2412.09269/) (2024)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Unraveling LLM Jailbreaks Through Safety Knowledge Neurons](https://mlsanity.com/paper/2509.01631/) (2025)
- [LLM Safety Alignment is Divergence Estimation in Disguise](https://mlsanity.com/paper/2502.00657/) (2025)
- [Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach](https://mlsanity.com/paper/2502.14100/) (2025)
- [The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context](https://mlsanity.com/paper/2504.02708/) (2025)
- [Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs](https://mlsanity.com/paper/2410.16327/) (2024)
- [CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations](https://mlsanity.com/paper/2507.06043/) (2025)
- [Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection](https://mlsanity.com/paper/2508.20766/) (2025)
- [Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Education](https://mlsanity.com/paper/2511.14423/) (2025)
- [Characterizing Selective Refusal Bias in Large Language Models](https://mlsanity.com/paper/2510.27087/) (2025)
- [Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization](https://mlsanity.com/paper/2511.19218/) (2025)
- [Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs](https://mlsanity.com/paper/2501.16534/) (2025)
- [Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization](https://mlsanity.com/paper/2505.04578/) (2025)
- [Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defense](https://mlsanity.com/paper/2606.05743/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. Although there are diverse jailbreak attack strategies, there is no unified understanding on why some methods succeed and others fail. This paper explores the behavior of harmful and harmless prompts in the LLM's representation space to investigate the intrinsic properties of successful jailbreak attacks. We hypothesize that successful attacks share some similar properties: They are effective in moving the representation of the harmful prompt towards the direction to the harmless prompts. We leverage hidden representations into the objective of existing jailbreak attacks to move the attacks along the acceptance direction, and conduct experiments to validate the above hypothesis using the proposed objective. We hope this study provides new insights into understanding how LLMs understand harmfulness information.