mlsanityData to 14 Oct 2026

Are LLMs Safe Beyond Text: Do Emojis Expose Gaps in Safety Evaluation

arXiv 2608.18164 ↗First posted 15 Aug 2026cs.CL cs.AI cs.CRcited by 0

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Safety evaluations of large language models (LLMs) predominantly rely on text-based adversarial prompts, potentially overlooking vulnerabilities arising from alternative input representations. This work examines emoji-augmented prompts as a test case for this gap, evaluating 50 prompts across four open-source LLMs (Mistral 7B, Qwen 2 7B, Gemma 2 9B, Llama 3 8B). Results show substantial variation in robustness: Gemma 2 9B and Mistral 7B exhibit non-zero success rates (10%), Llama 3 8B 6%, while Qwen 2 7B shows complete resistance (0% success rate). A chi-square test (chi2 = 32.94, p < 0.001) confirms significant differences in outcome distributions. These findings indicate that robustness is sensitive to input representation, and that evaluations restricted to standard text prompts may underrepresent model vulnerabilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 6 of its 11 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

6 of 11 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 5

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  3. 2016emoji2vec: Learning Emoji Representations from their Descriptioncited by 28
  4. 2025Emoti-Attack: Zero-Perturbation Adversarial Attacks on NLP Systems via Emoji Sequencescited by 5
  5. 2021Assessing Emoji Use in Modern Text Processing Toolscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Emoji-Based Jailbreaking of Large Language Models
  2. 2026Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs
  3. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
  4. 2026SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs
  5. 2024Robustness of Large Language Models Against Adversarial Attacks
  6. 2024Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?
  7. 2026IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
  8. 2025Safety and Security Analysis of Large Language Models: Benchmarking Risk Profile and Harm Potential