# Realistic Evaluation of Toxicity in Large Language Models

arXiv: https://arxiv.org/abs/2405.10659 · HTML: https://mlsanity.com/paper/2405.10659/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 17 May 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)

## Built on by (most cited first, 15 of 15)

- [Head Pursuit: Probing Attention Specialization in Multimodal Transformers](https://mlsanity.com/paper/2510.21518/) (2025)
- [LinEAS: End-to-end Learning of Activation Steering with a Distributional Loss](https://mlsanity.com/paper/2503.10679/) (2025)
- [TAI3: Testing Agent Integrity in Interpreting User Intent](https://mlsanity.com/paper/2506.07524/) (2025)
- [PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platforms](https://mlsanity.com/paper/2410.05315/) (2024)
- [Large Language Model Sourcing: A Survey](https://mlsanity.com/paper/2510.10161/) (2025)
- [MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning](https://mlsanity.com/paper/2506.16792/) (2025)
- [Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directions](https://mlsanity.com/paper/2509.25539/) (2025)
- [Evaluating Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards](https://mlsanity.com/paper/2506.04975/) (2025)
- [Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Models](https://mlsanity.com/paper/2509.15478/) (2025)
- [ToxiTwitch: Toward Emote-Aware Hybrid Moderation for Live Streaming Platforms](https://mlsanity.com/paper/2601.15605/) (2026)
- [DocNet: Semantic Structure in Inductive Bias Detection Models](https://mlsanity.com/paper/2406.10965/) (2024)
- [Stated Preference for Interaction and Continued Engagement (SPICE): Evaluating an LLM's Willingness to Re-engage in Conversation](https://mlsanity.com/paper/2509.09043/) (2025)
- [Remembering Unequally: Global and Disciplinary Bias in LLM Reconstruction of Scholarly Coauthor Lists](https://mlsanity.com/paper/2511.00476/) (2025)
- [Measuring and Mitigating Toxicity in Large Language Models: A Comprehensive Replication Study](https://mlsanity.com/paper/2605.14087/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have become integral to our professional workflows and daily lives. Nevertheless, these machine companions of ours have a critical flaw: the huge amount of data which endows them with vast and diverse knowledge, also exposes them to the inevitable toxicity and bias. While most LLMs incorporate defense mechanisms to prevent the generation of harmful content, these safeguards can be easily bypassed with minimal prompt engineering. In this paper, we introduce the new Thoroughly Engineered Toxicity (TET) dataset, comprising manually crafted prompts designed to nullify the protective layers of such models. Through extensive evaluations, we demonstrate the pivotal role of TET in providing a rigorous benchmark for evaluation of toxicity awareness in several popular LLMs: it highlights the toxicity in the LLMs that might remain hidden when using normal prompts, thus revealing subtler issues in their behavior.