mlsanityData to 13 Oct 2026

Unnatural Languages Are Not Bugs but Features for LLMs

arXiv 2503.01926 ↗First posted 2 Mar 2025cs.CL cs.AIcited by 4

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) have been observed to process non-human-readable text sequences, such as jailbreak prompts, often viewed as a bug for aligned LLMs. In this work, we present a systematic investigation challenging this perception, demonstrating that unnatural languages - strings that appear incomprehensible to humans but maintain semantic meanings for LLMs - contain latent features usable by models. Notably, unnatural languages possess latent features that can be generalized across different models and tasks during inference. Furthermore, models fine-tuned on unnatural versions of instruction datasets perform on-par with those trained on natural language, achieving 49.71 win rates in Length-controlled AlpacaEval 2.0 in average across various base models. In addition, through comprehensive analysis, we demonstrate that LLMs process unnatural languages by filtering noise and inferring contextual meaning from filtered words.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 29 of its 40 references that are in the map. First posted on arXiv 2 Mar 2025.

Builds on

29 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2024The Llama 3 Herd of Modelscited by 14,486
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  8. 2013Intriguing properties of neural networkscited by 5,137
  9. 2024GPT-4o System Cardcited by 4,536
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  11. 2023Mistral 7Bcited by 3,123
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
14 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,968
  2. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  3. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  4. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 718
  5. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 444
  6. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 280
  7. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  8. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 188
  9. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  10. 2020UnNatural Language Inferencecited by 68
  11. 2024Mission: Impossible Language Modelscited by 31
  12. 2019Efficient Saliency Maps for Explainable AIcited by 30
  13. 2024Instruction Following without Instruction Tuningcited by 11
  14. 2023Unnatural language processing: How do language models handle machine-generated prompts?cited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 92
  2. 2025Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?cited by 6
  3. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  4. 2026Is Human-Readable Text Necessary for Effective LLM Fine-Tuning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs' Understanding of Natural Language Revealed
  2. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  3. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
  4. 2026Understanding Large Language Models
  5. 2025Large Language Model probabilities cannot distinguish between possible and impossible language
  6. 2024Impact of Non-Standard Unicode Characters on Security and Comprehension in Large Language Models
  7. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models
  8. 2026The Implications of Linguistic Illegibility for LLM Security