# Are LLMs Safe Beyond Text: Do Emojis Expose Gaps in Safety Evaluation

arXiv: https://arxiv.org/abs/2608.18164 · HTML: https://mlsanity.com/paper/2608.18164/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 15 Aug 2026.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 6 of 6)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [emoji2vec: Learning Emoji Representations from their Description](https://mlsanity.com/paper/1609.08359/) (2016)
- [Emoti-Attack: Zero-Perturbation Adversarial Attacks on NLP Systems via Emoji Sequences](https://mlsanity.com/paper/2502.17392/) (2025)
- [Assessing Emoji Use in Modern Text Processing Tools](https://mlsanity.com/paper/2101.00430/) (2021)

## Abstract (arXiv)

Safety evaluations of large language models (LLMs) predominantly rely on text-based adversarial prompts, potentially overlooking vulnerabilities arising from alternative input representations. This work examines emoji-augmented prompts as a test case for this gap, evaluating 50 prompts across four open-source LLMs (Mistral 7B, Qwen 2 7B, Gemma 2 9B, Llama 3 8B). Results show substantial variation in robustness: Gemma 2 9B and Mistral 7B exhibit non-zero success rates (10%), Llama 3 8B 6%, while Qwen 2 7B shows complete resistance (0% success rate). A chi-square test (chi2 = 32.94, p < 0.001) confirms significant differences in outcome distributions. These findings indicate that robustness is sensitive to input representation, and that evaluations restricted to standard text prompts may underrepresent model vulnerabilities.