mlsanityData to 7 Oct 2026

Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift

arXiv 2509.06338 ↗First posted 8 Sep 2025cs.CR cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The widespread distribution of Large Language Models (LLMs) through public platforms like Hugging Face introduces significant security challenges. While these platforms perform basic security scans, they often fail to detect subtle manipulations within the embedding layer. This work identifies a novel class of deployment phase attacks that exploit this vulnerability by injecting imperceptible perturbations directly into the embedding layer outputs without modifying model weights or input text. These perturbations, though statistically benign, systematically bypass safety alignment mechanisms and induce harmful behaviors during inference. We propose Search based Embedding Poisoning(SEP), a practical, model agnostic framework that introduces carefully optimized perturbations into embeddings associated with high risk tokens. SEP leverages a predictable linear transition in model responses, from refusal to harmful output to semantic deviation to identify a narrow perturbation window that evades alignment safeguards. Evaluated across six aligned LLMs, SEP achieves an average attack success rate of 96.43% while preserving benign task performance and evading conventional detection mechanisms. Our findings reveal a critical oversight in deployment security and emphasize the urgent need for embedding level integrity checks in future LLM defense strategies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 40 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

23 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation8 Jul
  2. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  3. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignment20 May
  4. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  7. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  10. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  11. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  12. 2024Exploiting LLM Quantizationcited by 61
4 more
  1. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  2. 2024ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddingscited by 31
  3. 2024Improved Large Language Model Jailbreak Detection via Pretrained Embeddingscited by 10
  4. 2025Smoothed Embeddings for Robust Language Modelscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SEA: Low-Resource Safety Alignment for Multimodal Large Language Models via Synthetic Embeddings
  2. 2025Semantic Representation Attack against Aligned Large Language Models
  3. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
  4. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack
  5. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
  6. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  7. 2026Faithful Dual-constrained Erasure for Robust LLM Safety Alignment
  8. 2026Safety Is Not Universal: The Selective Safety Trap in LLM Alignment