mlsanityData to 7 Oct 2026

Adversarial Tokenization

arXiv 2503.02174 ↗First posted 4 Mar 2025cs.CL cs.AI cs.LGcited by 11

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Current LLM pipelines account for only one possible tokenization for a given string, ignoring exponentially many alternative tokenizations during training and inference. For example, the standard Llama3 tokenization of penguin is [p,enguin], yet [peng,uin] is another perfectly valid alternative. In this paper, we show that despite LLMs being trained solely on one tokenization, they still retain semantic understanding of other tokenizations, raising questions about their implications in LLM safety. Put succinctly, we answer the following question: can we adversarially tokenize an obviously malicious string to evade safety and alignment restrictions? We show that not only is adversarial tokenization an effective yet previously neglected axis of attack, but it is also competitive against existing state-of-the-art adversarial approaches without changing the text of the harmful request. We empirically validate this exploit across three state-of-the-art LLMs and adversarial datasets, revealing a previously unknown vulnerability in subword models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 38 of its 55 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

38 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
  2. 2024Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks1 Jul
  3. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models13 Jun
  4. 2024Distract Large Language Models for Automatic Jailbreak Attack13 Mar
  5. 2024ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings25 Feb
  6. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
17 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  4. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  5. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  6. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  7. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  8. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  9. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  10. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
  11. 2024From Language Models over Tokens to Language Models over Characterscited by 28
  12. 2024Where is the signal in tokenization space?cited by 21
  13. 2023Should you marginalize over possible tokenizations?cited by 17
  14. 2023Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologiescited by 16
  15. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  16. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12
  17. 2025MBTSAD: Mitigating Backdoors in Language Models Based on Token Splitting and Attention Distillationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Security Concerns for Large Language Models: A Surveycited by 26
  2. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  3. 2025Auditing Pay-Per-Token in Large Language Modelscited by 4
  4. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  5. 2025UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8cited by 3
  6. 2026Proxy Compression for Language Modelingcited by 3
  7. 2026Stochasticity in Tokenisation Improves Robustnesscited by 1
  8. 2025Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-servicecited by 0
  9. 2026Evolving Jailbreaks: Automated Multi-Objective Long-Tail Attacks on Large Language Modelscited by 0
  10. 2026Emergent retokenization symmetry in large language models: phenomenology and applicationscited by 0
  11. 2026The Tokens Remember: When Tokenization Bypasses Knowledge Editing and Unlearningcited by 0