mlsanityData to 7 Oct 2026

Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries

arXiv 2405.20653 ↗First posted 31 May 2024cs.AIcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have led to impressive alignment where models learn to distinguish harmful from harmless queries through supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF). In this paper, we reveal a subtle yet impactful weakness in these aligned models. We find that simply appending multiple end of sequence (eos) tokens can cause a phenomenon we call context segmentation, which effectively shifts both harmful and benign inputs closer to the refusal boundary in the hidden space. Building on this observation, we propose a straightforward method to BOOST jailbreak attacks by appending eos tokens. Our systematic evaluation shows that this strategy significantly increases the attack success rate across 8 representative jailbreak techniques and 16 open-source LLMs, ranging from 2B to 72B parameters. Moreover, we develop a novel probing mechanism for commercial APIs and discover that major providers such as OpenAI, Anthropic, and Qwen do not filter eos tokens, making them similarly vulnerable. These findings highlight a hidden yet critical blind spot in existing alignment and content filtering approaches. We call for heightened attention to eos tokens' unintended influence on model behaviors, particularly in production systems. Our work not only calls for an input-filtering based defense, but also points to new defenses that make refusal boundaries more robust and generalizable, as well as fundamental alignment techniques that can defend against context segmentation attacks.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 61 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

41 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Response22 May
  2. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  3. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  5. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  7. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  8. 2023Are aligned neural networks adversarially aligned?26 Jun
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
20 more
  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  7. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  8. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  9. 2024A StrongREJECT for Empty Jailbreakscited by 352
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  12. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  13. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  14. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  15. 2023Assessing Prompt Injection Risks in 200+ Custom GPTscited by 49
  16. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  17. 2023Large Language Models for Code Analysis: Do LLMs Really Do Their Job?cited by 44
  18. 2023A Latent Space Theory for Emergent Abilities in Large Language Modelscited by 42
  19. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  20. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 10; 2026: 6.

26 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2024PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMscited by 17
  4. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  5. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  6. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  7. 2025DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modificationcited by 10
  8. 2024Decoupled Alignment for Robust Plug-and-Play Adaptationcited by 9
  9. 2024Advanced AI Framework for Enhanced Detection and Assessment of Abdominal Trauma: Integrating 3D Segmentation with 2D CNN and RNN Modelscited by 9
  10. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  11. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
  12. 2024Fast John Ellipsoid Computation with Differential Privacy Optimizationcited by 7
14 more
  1. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 5
  2. 2024Global Challenge for Safe and Secure LLMs Track 1cited by 4
  3. 2025Silent Tokens, Loud Effects: Padding in LLMscited by 4
  4. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  5. 2026State-Dependent Safety Failures in Multi-Turn Language Model Interactioncited by 4
  6. 2024Differentially Private Kernel Density Estimationcited by 2
  7. 2025Effective and Efficient Jailbreaks of Black-Box LLMs with Cross-Behavior Attackscited by 2
  8. 2025Rethinking Toxicity Evaluation in Large Language Models: A Multi-Label Perspectivecited by 2
  9. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  10. 2026Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Modelscited by 2
  11. 2024Soft-Label Integration for Robust Toxicity Classificationcited by 1
  12. 2026Jailbreaking Large Language Models through Iterative Tool-Disguised Attacks via Reinforcement Learningcited by 1
  13. 2026TrojanPraise: Jailbreak LLMs via Benign Fine-Tuningcited by 1
  14. 2026RoguePrompt: Dual-Layer Encoding for Self-Reconstruction to Circumvent LLM Moderationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
  2. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off
  3. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  4. 2024Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models
  5. 2025Improving LLM Safety Alignment with Dual-Objective Optimization
  6. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
  7. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
  8. 2026Fail-Closed Alignment for Large Language Models