# Towards Robust Toxic Content Classification

arXiv: https://arxiv.org/abs/1912.06872 · HTML: https://mlsanity.com/paper/1912.06872/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2019.

## Built on by (most cited first, 22 of 22)

- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](https://mlsanity.com/paper/2202.11176/) (2022)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [Benchmarking LLM Guardrails in Handling Multilingual Toxicity](https://mlsanity.com/paper/2410.22153/) (2024)
- [Automated Identification of Toxic Code Reviews Using ToxiCR](https://mlsanity.com/paper/2202.13056/) (2022)
- [Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators](https://mlsanity.com/paper/2411.05854/) (2024)
- [Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI](https://mlsanity.com/paper/2301.01874/) (2023)
- [Cyberbullying Classifiers are Sensitive to Model-Agnostic Perturbations](https://mlsanity.com/paper/2201.06384/) (2022)
- [When Vision Fails: Text Attacks Against ViT and OCR](https://mlsanity.com/paper/2306.07033/) (2023)
- [A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity](https://mlsanity.com/paper/2311.04345/) (2023)
- [Misspellings in Natural Language Processing: A survey](https://mlsanity.com/paper/2501.16836/) (2025)
- [SafeGPT: Preventing Data Leakage and Unethical Outputs in Enterprise LLM Use](https://mlsanity.com/paper/2601.06366/) (2026)
- [Cisco at SemEval-2021 Task 5: What's Toxic?: Leveraging Transformers for Multiple Toxic Span Extraction from Online Comments](https://mlsanity.com/paper/2105.13959/) (2021)
- [Don't be a Fool: Pooling Strategies in Offensive Language Detection from User-Intended Adversarial Attacks](https://mlsanity.com/paper/2403.15467/) (2024)
- [Decentralised Moderation for Interoperable Social Networks: A Conversation-based Approach for Pleroma and the Fediverse](https://mlsanity.com/paper/2404.03048/) (2024)
- [Combating Toxic Language: A Review of LLM-Based Strategies for Software Engineering](https://mlsanity.com/paper/2504.15439/) (2025)
- [MDIT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Models](https://mlsanity.com/paper/2505.17144/) (2025)
- [BinaryPPO: Efficient Policy Optimization for Binary Classification](https://mlsanity.com/paper/2602.02708/) (2026)
- [ContiGuard: A Framework for Continual Toxicity Detection Against Evolving Evasive Perturbations](https://mlsanity.com/paper/2603.14843/) (2026)
- [What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks](https://mlsanity.com/paper/2606.09700/) (2026)
- [Aligned in Form, Not in Meaning: The Comprehension - Containment Decoupling of LLM Safety in Low-Resource Bangla Derogatory Speech](https://mlsanity.com/paper/2608.02941/) (2026)

## Abstract (arXiv)

Toxic content detection aims to identify content that can offend or harm its recipients. Automated classifiers of toxic content need to be robust against adversaries who deliberately try to bypass filters. We propose a method of generating realistic model-agnostic attacks using a lexicon of toxic tokens, which attempts to mislead toxicity classifiers by diluting the toxicity signal either by obfuscating toxic tokens through character-level perturbations, or by injecting non-toxic distractor tokens. We show that these realistic attacks reduce the detection recall of state-of-the-art neural toxicity detectors, including those using ELMo and BERT, by more than 50% in some cases. We explore two approaches for defending against such attacks. First, we examine the effect of training on synthetically noised data. Second, we propose the Contextual Denoising Autoencoder (CDAE): a method for learning robust representations that uses character-level and contextual information to denoise perturbed tokens. We show that the two approaches are complementary, improving robustness to both character-level perturbations and distractors, recovering a considerable portion of the lost accuracy. Finally, we analyze the robustness characteristics of the most competitive methods and outline practical considerations for improving toxicity detectors.