mlsanityData to 14 Oct 2026

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection

arXiv 2411.01077 ↗First posted 1 Nov 2024cs.CL cs.LGcited by 23

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreaking techniques trick Large Language Models (LLMs) into producing restricted output, posing a potential threat. One line of defense is to use another LLM as a Judge to evaluate the harmfulness of generated text. However, we reveal that these Judge LLMs are vulnerable to token segmentation bias, an issue that arises when delimiters alter the tokenization process, splitting words into smaller sub-tokens. This alters the embeddings of the entire sequence, reducing detection accuracy and allowing harmful content to be misclassified as safe. In this paper, we introduce Emoji Attack, a novel strategy that amplifies existing jailbreak prompts by exploiting token segmentation bias. Our method leverages in-context learning to systematically insert emojis into text before it is evaluated by a Judge LLM, inducing embedding distortions that significantly lower the likelihood of detecting unsafe content. Unlike traditional delimiters, emojis also introduce semantic ambiguity, making them particularly effective in this attack. Through experiments on state-of-the-art Judge LLMs, we demonstrate that Emoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 37 of its 46 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

37 of 46 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework11 Oct
  2. 2024Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization15 May
  3. 2024Boosting Jailbreak Attack with Momentum2 May
  4. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  6. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models18 Mar
  7. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  8. 2024Query-Based Adversarial Prompt Generation19 Feb
  9. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  11. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  12. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
6 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  4. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  12. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
7 more
  1. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  2. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  3. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  4. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  5. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  6. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 51
  7. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 12.

23 papers.

Sort:

By area:

  1. 2025Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systemscited by 26
  2. 2026Efficient and accurate steering of Large Language Models through attention-guided feature learningcited by 8
  3. 2025AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Modelscited by 7
  4. 2025Adversarial D'ej`a Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attackscited by 7
  5. 2026Security in LLM-as-a-Judge: A Comprehensive SoKcited by 5
  6. 2025MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMscited by 4
  7. 2026Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judgescited by 4
  8. 2025MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuningcited by 3
  9. 2026MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attackscited by 3
  10. 2025When Smiley Turns Hostile: Interpreting How Emojis Trigger LLMs' Toxicitycited by 2
  11. 2025Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approachcited by 2
  12. 2026Jailbreaking Large Language Models through Iterative Tool-Disguised Attacks via Reinforcement Learningcited by 2
11 more
  1. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  2. 2026False Friends in the Shell: Unveiling the Emoticon Semantic Confusion in Large Language Modelscited by 1
  3. 2026ShallowJail: Steering Jailbreaks against Large Language Modelscited by 1
  4. 2026D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewritingcited by 1
  5. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignmentcited by 0
  6. 2025Obfuscation Rules for Detecting and Detoxifying Korean Toxicitycited by 0
  7. 2025How to Trick Your AI TA: A Systematic Study of Academic Jailbreaking in LLM Code Evaluationcited by 0
  8. 2026Jailbreaking LLMs via Calibrationcited by 0
  9. 2026SelfGrader: LLM Jailbreak Detection via Anchored Token-Level Logitscited by 0
  10. 2026DeepSeek Robustness Against Semantic-Character Dual-Space Mutated Prompt Injectioncited by 0
  11. 2026CS-Guard: Benchmarking LLM Guardrails for Code Generation Securitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Emoji-Based Jailbreaking of Large Language Models
  2. 2024Improved Large Language Model Jailbreak Detection via Pretrained Embeddings
  3. 2024Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models
  4. 2025Machine Learning for Detection and Analysis of Novel LLM Jailbreaks
  5. 2023JailGuard: A Universal Detection Framework for LLM Prompt-based Attacks
  6. 2025Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Is
  7. 2024Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey