mlsanityData to 7 Oct 2026

Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models

arXiv 2501.18280 ↗First posted 30 Jan 2025cs.CL cs.AI cs.LGcited by 5

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean. Inspired by this observation, we propose novel, efficient methods to search for **universal magic words** that attack text embedding models. Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards. Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words. Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages. To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 50 of its 59 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

50 of 59 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  4. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  5. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak7 Dec
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  7. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  8. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  9. 2023Self-Guard: Empower the LLM to Safeguard Itself24 Oct
  10. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  12. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
8 more
  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  2. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  3. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  4. 2023Detecting Language Model Attacks with Perplexity27 Aug
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models17 Jul
  8. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  10. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  11. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
18 more
  1. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  2. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  3. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  4. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  5. 2023Improving Text Embeddings with Large Language Modelscited by 295
  6. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  7. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  8. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  9. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  10. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  11. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  12. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  13. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103
  14. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  15. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  16. 2023Robust Safety Classifier for Large Language Models: Adversarial Prompt Shieldcited by 24
  17. 2023Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principlescited by 17
  18. 2025Phase Transitions in Large Language Models and the O(N) Modelcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuationcited by 2
  2. 2025Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEBcited by 1
  3. 2026ShallowJail: Steering Jailbreaks against Large Language Modelscited by 1
  4. 2025Exploiting Latent Space Discontinuities for Building Universal LLM Jailbreaks and Data Extraction Attackscited by 0
  5. 2026Adaptive Prompt Embedding Optimization for LLM Jailbreakingcited by 0