mlsanityData to 14 Oct 2026

Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMs

arXiv 2505.14226 ↗First posted 20 May 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Safety-aligned LLMs remain vulnerable to digital phenomena like textese that introduce non-canonical perturbations to words but preserve the phonetics. We introduce CMP-RT (code-mixed phonetic perturbations for red-teaming), a novel diagnostic probe that pinpoints tokenization as the root cause of this vulnerability. A mechanistic analysis reveals that phonetic perturbations fragment safety-critical tokens into benign sub-words, suppressing their attribution scores while preserving prompt interpretability -- causing safety mechanisms to fail despite excellent input understanding. We demonstrate that this vulnerability evades standard defenses, persists across modalities and state-of-the-art (SOTA) models including Gemini-3-Pro, and scales through simple supervised fine-tuning (SFT). Furthermore, layer-wise probing shows perturbed and canonical input representations align up to a critical layer depth; enforcing output equivalence robustly recovers the lost representations, providing causal evidence for a structural gap between pre-training and alignment, and establishing tokenization as a critical, under-examined vulnerability in current safety pipelines.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 32 of its 58 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

32 of 58 references.

References · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2023Mistral 7Bcited by 3,164
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
20 more
  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  2. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  3. 2023Detecting Language Model Attacks with Perplexitycited by 431
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  5. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  6. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  7. 2024Best-of-N Jailbreakingcited by 76
  8. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  9. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  10. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 64
  11. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Modelscited by 59
  12. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 42
  13. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  14. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 27
  15. 2024Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Modelscited by 25
  16. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  17. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20
  18. 2024Sandwich attack: Multi-language Mixture Adaptive Attack on LLMscited by 20
  19. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systemscited by 14
  20. 2024Towards Red Teaming in Multimodal and Multilingual Translationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Surface Forms: A Comprehensive, Mechanism-Oriented Taxonomy of Indirect Linguistic Encoding for LLM-Based Coded Language Detectioncited by 0
  2. 2026Temporally-Resolved Token Attribution Reveals the Generation Dynamics of Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment
  2. 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs
  3. 2026How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
  4. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  5. 2026SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech
  6. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  7. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  8. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance