# Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech

arXiv: https://arxiv.org/abs/2109.00591 · HTML: https://mlsanity.com/paper/2109.00591/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Contextualizing Hate Speech Classifiers with Post-hoc Explanation](https://mlsanity.com/paper/2005.02439/) (2020)
- [Towards Hate Speech Detection at Large via Deep Generative Modeling](https://mlsanity.com/paper/2005.06370/) (2020)
- [UTFPR at SemEval-2019 Task 5: Hate Speech Identification with Recurrent Neural Networks](https://mlsanity.com/paper/1904.07839/) (2019)

## Built on by (most cited first, 13 of 13)

- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information](https://mlsanity.com/paper/2409.14740/) (2024)
- [HateBench: Benchmarking Hate Speech Detectors on LLM-Generated Content and Hate Campaigns](https://mlsanity.com/paper/2501.16750/) (2025)
- [Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2303.02513/) (2023)
- [Character-level HyperNetworks for Hate Speech Detection](https://mlsanity.com/paper/2111.06336/) (2021)
- [Generative AI for Hate Speech Detection: Evaluation and Findings](https://mlsanity.com/paper/2311.09993/) (2023)
- [CRUSH: Contextually Regularized and User anchored Self-supervised Hate speech Detection](https://mlsanity.com/paper/2204.06389/) (2022)
- [Going Extreme: Comparative Analysis of Hate Speech in Parler and Gab](https://mlsanity.com/paper/2201.11770/) (2022)
- [Detecting Multidimensional Political Incivility on Social Media](https://mlsanity.com/paper/2305.14964/) (2023)
- [Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledge](https://mlsanity.com/paper/2311.14865/) (2023)
- [A Target-Aware Analysis of Data Augmentation for Hate Speech Detection](https://mlsanity.com/paper/2410.08053/) (2024)
- [From Measurement Instruments to Data: Leveraging Theory-Driven Synthetic Training Data for Classifying Social Constructs](https://mlsanity.com/paper/2410.12622/) (2024)

## Abstract (arXiv)

Automatic hate speech detection is hampered by the scarcity of labeled datasetd, leading to poor generalization. We employ pretrained language models (LMs) to alleviate this data bottleneck. We utilize the GPT LM for generating large amounts of synthetic hate speech sequences from available labeled examples, and leverage the generated data in fine-tuning large pretrained LMs on hate detection. An empirical study using the models of BERT, RoBERTa and ALBERT, shows that this approach improves generalization significantly and consistently within and across data distributions. In fact, we find that generating relevant labeled hate speech sequences is preferable to using out-of-domain, and sometimes also within-domain, human-labeled examples.