# Towards Inclusive Toxic Content Moderation: Addressing Vulnerabilities to Adversarial Attacks in Toxicity Classifiers Tackling LLM-generated Content

arXiv: https://arxiv.org/abs/2509.12672 · HTML: https://mlsanity.com/paper/2509.12672/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Gradient-based Adversarial Attacks against Text Transformers](https://mlsanity.com/paper/2104.13733/) (2021)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [MarkLLM: An Open-Source Toolkit for LLM Watermarking](https://mlsanity.com/paper/2405.10051/) (2024)
- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- ["Call me sexist, but...": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples](https://mlsanity.com/paper/2004.12764/) (2020)
- [HowkGPT: Investigating the Detection of ChatGPT-generated University Student Homework through Context-Aware Perplexity Analysis](https://mlsanity.com/paper/2305.18226/) (2023)
- [The Rise of AI-Generated Content in Wikipedia](https://mlsanity.com/paper/2410.08044/) (2024)
- [Towards Building a Robust Toxicity Predictor](https://mlsanity.com/paper/2404.08690/) (2024)
- [Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability](https://mlsanity.com/paper/2407.19842/) (2024)
- [Token-Modification Adversarial Attacks for Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.00676/) (2021)
- [A Critical Reflection on the Use of Toxicity Detection Algorithms in Proactive Content Moderation Systems](https://mlsanity.com/paper/2401.10629/) (2024)
- [A Classification-Guided Approach for Adversarial Attacks against Neural Machine Translation](https://mlsanity.com/paper/2308.15246/) (2023)
- [Toxic language detection: a systematic review of Arabic datasets](https://mlsanity.com/paper/2312.07228/) (2023)
- [Enhancing Adversarial Text Attacks on BERT Models with Projected Gradient Descent](https://mlsanity.com/paper/2407.21073/) (2024)

## Abstract (arXiv)

Adversarial perturbations can reduce state-of-the-art toxicity classifiers to near-zero accuracy, yet existing defences treat models as black boxes. We apply mechanistic interpretability to toxicity classification for the first time, identifying the internal attention-head circuits responsible for both correct classification and adversarial vulnerability. Across a 2×2 factorial study (BERT × RoBERTa) × (Jigsaw × ToxiGen), extended to Llama Guard~2 (8B), we show that zeroing a single attention head recovers up to 70.4 pp of adversarial accuracy for RoBERTa on Jigsaw and 37.3 pp for Llama Guard 2 on ToxiGen, at ≤0.6 pp clean cost. Vulnerable heads generalise to held-out examples within ≤1 pp, and a class-imbalance sweep confirms they act as selective toxic-class detectors. Head suppression matches or outperforms adversarial training on Jigsaw; data augmentation dominates on ToxiGen: a dataset-specific reversal explained by whether the classifier encodes a concentrated bottleneck or a distributed circuit. Demographic analysis across 20 Jigsaw and 13 ToxiGen minority groups reveals structurally unequal adversarial vulnerability, exposing mechanistically traceable fairness gaps in current toxicity classifiers.