# Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering

arXiv: https://arxiv.org/abs/2211.03433 · HTML: https://mlsanity.com/paper/2211.03433/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 27 of 27)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://mlsanity.com/paper/1909.04251/) (2019)
- [CONAN -- COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://mlsanity.com/paper/1910.03270/) (2019)
- [Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech](https://mlsanity.com/paper/2107.08720/) (2021)
- [Thou shalt not hate: Countering Online Hate Speech](https://mlsanity.com/paper/1808.04409/) (2018)
- [Demoting Racial Bias in Hate Speech Detection](https://mlsanity.com/paper/2005.12246/) (2020)
- [Generating Counter Narratives against Online Hate Speech: Data and Strategies](https://mlsanity.com/paper/2004.04216/) (2020)
- [Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective](https://mlsanity.com/paper/2012.12305/) (2020)
- [Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech](https://mlsanity.com/paper/2106.01625/) (2021)
- [Towards Knowledge-Grounded Counter Narrative Generation for Hate Speech](https://mlsanity.com/paper/2106.11783/) (2021)
- [Like trainer, like bot? Inheritance of bias in algorithmic content moderation](https://mlsanity.com/paper/1707.01477/) (2017)
- [Hate Speech and Counter Speech Detection: Conversational Context Does Matter](https://mlsanity.com/paper/2206.06423/) (2022)
- [Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Study](https://mlsanity.com/paper/2204.01440/) (2022)
- [DeepHate: Hate Speech Detection via Multi-Faceted Text Representations](https://mlsanity.com/paper/2103.11799/) (2021)
- [Augmenting Neural Response Generation with Context-Aware Topical Attention](https://mlsanity.com/paper/1811.01063/) (2018)
- [Analyzing the hate and counter speech accounts on Twitter](https://mlsanity.com/paper/1812.02712/) (2018)
- [ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning](https://mlsanity.com/paper/2105.12995/) (2021)
- [Empowering NGOs in Countering Online Hate Messages](https://mlsanity.com/paper/2107.02472/) (2021)
- [Detection of Cyberbullying Incidents on the Instagram Social Network](https://mlsanity.com/paper/1503.03909/) (2015)

## Built on by (most cited first, 30 of 40)

- [Navigating the OverKill in Large Language Models](https://mlsanity.com/paper/2401.17633/) (2024)
- [Using In-Context Learning to Improve Dialogue Safety](https://mlsanity.com/paper/2302.00871/) (2023)
- [Beyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language](https://mlsanity.com/paper/2311.00161/) (2023)
- [NLP for Counterspeech against Hate: A Survey and How-To Guide](https://mlsanity.com/paper/2403.20103/) (2024)
- [NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps](https://mlsanity.com/paper/2404.01651/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Understanding Counterspeech for Online Harm Mitigation](https://mlsanity.com/paper/2307.04761/) (2023)
- [DisCGen: A Framework for Discourse-Informed Counterspeech Generation](https://mlsanity.com/paper/2311.18147/) (2023)
- [CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network](https://mlsanity.com/paper/2303.03387/) (2023)
- [Outcome-Constrained Large Language Models for Countering Hate Speech](https://mlsanity.com/paper/2403.17146/) (2024)
- [Is Safer Better? The Impact of Guardrails on the Argumentative Strength of LLMs in Hate Speech Countering](https://mlsanity.com/paper/2410.03466/) (2024)
- [Weigh Your Own Words: Improving Hate Speech Counter Narrative Generation via Attention Regularization](https://mlsanity.com/paper/2309.02311/) (2023)
- [Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?](https://mlsanity.com/paper/2305.15908/) (2023)
- [A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models](https://mlsanity.com/paper/2402.11676/) (2024)
- [A LLM-Based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation](https://mlsanity.com/paper/2406.15227/) (2024)
- [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024)
- [Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?](https://mlsanity.com/paper/2307.07047/) (2023)
- [Echoes of Discord: Forecasting Hater Reactions to Counterspeech](https://mlsanity.com/paper/2501.16235/) (2025)
- [CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs](https://mlsanity.com/paper/2501.17581/) (2025)
- [Consolidating Strategies for Countering Hate Speech Using Persuasive Dialogues](https://mlsanity.com/paper/2401.07810/) (2024)
- [Hatred Stems from Ignorance! Distillation of the Persuasion Modes in Countering Conversational Hate Speech](https://mlsanity.com/paper/2403.15449/) (2024)
- [Debunking with Dialogue? Exploring AI-Generated Counterspeech to Challenge Conspiracy Theories](https://mlsanity.com/paper/2504.16604/) (2025)
- [Counterspeech the ultimate shield! Multi-Conditioned Counterspeech Generation through Attributed Prefix Learning](https://mlsanity.com/paper/2505.11958/) (2025)
- [Beating Harmful Stereotypes Through Facts: RAG-based Counter-speech Generation](https://mlsanity.com/paper/2510.12316/) (2025)
- [METRO: Towards Strategy Induction from Expert Dialogue Transcripts for Non-collaborative Dialogues](https://mlsanity.com/paper/2604.11427/) (2026)
- [Alternative Speech: Complementary Method to Counter-Narrative for Better Discourse](https://mlsanity.com/paper/2401.14616/) (2024)
- [Causal Inference for Human-Language Model Collaboration](https://mlsanity.com/paper/2404.00207/) (2024)
- [Think Like a Person Before Responding: A Multi-Faceted Evaluation of Persona-Guided LLMs for Countering Hate](https://mlsanity.com/paper/2506.04043/) (2025)
- [EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Norms](https://mlsanity.com/paper/2507.20264/) (2025)
- [Hate Speech Targets Detection in Parler using BERT](https://mlsanity.com/paper/2304.01179/) (2023)

## Abstract (arXiv)

Fighting online hate speech is a challenge that is usually addressed using Natural Language Processing via automatic detection and removal of hate content. Besides this approach, counter narratives have emerged as an effective tool employed by NGOs to respond to online hate on social media platforms. For this reason, Natural Language Generation is currently being studied as a way to automatize counter narrative writing. However, the existing resources necessary to train NLG models are limited to 2-turn interactions (a hate speech and a counter narrative as response), while in real life, interactions can consist of multiple turns. In this paper, we present a hybrid approach for dialogical data collection, which combines the intervention of human expert annotators over machine generated dialogues obtained using 19 different configurations. The result of this work is DIALOCONAN, the first dataset comprising over 3000 fictitious multi-turn dialogues between a hater and an NGO operator, covering 6 targets of hate.