# Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts

arXiv: https://arxiv.org/abs/2108.11830 · HTML: https://mlsanity.com/paper/2108.11830/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 63 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 30 of 38)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://mlsanity.com/paper/2004.08449/) (2020)
- [Towards Controllable Biases in Language Generation](https://mlsanity.com/paper/2005.00268/) (2020)
- [CONAN -- COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://mlsanity.com/paper/1910.03270/) (2019)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)

## Built on by (most cited first, 30 of 56)

- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [COLD: A Benchmark for Chinese Offensive Language Detection](https://mlsanity.com/paper/2201.06025/) (2022)
- [On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark](https://mlsanity.com/paper/2110.08466/) (2021)
- [Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models](https://mlsanity.com/paper/2202.04173/) (2022)
- [Mix and Match: Learning-free Controllable Text Generation using Energy Language Models](https://mlsanity.com/paper/2203.13299/) (2022)
- [A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects](https://mlsanity.com/paper/2305.02750/) (2023)
- [BiasAsker: Measuring the Bias in Conversational AI System](https://mlsanity.com/paper/2305.12434/) (2023)
- [Using In-Context Learning to Improve Dialogue Safety](https://mlsanity.com/paper/2302.00871/) (2023)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks](https://mlsanity.com/paper/2202.08011/) (2022)
- ["They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversations](https://mlsanity.com/paper/2405.05378/) (2024)
- [Robust Conversational Agents against Imperceptible Toxicity Triggers](https://mlsanity.com/paper/2205.02392/) (2022)
- [MentalManip: A Dataset For Fine-grained Analysis of Mental Manipulation in Conversations](https://mlsanity.com/paper/2405.16584/) (2024)
- [Granular Privacy Control for Geolocation with Vision Language Models](https://mlsanity.com/paper/2407.04952/) (2024)
- [PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models](https://mlsanity.com/paper/2405.09373/) (2024)
- [Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models](https://mlsanity.com/paper/2305.14718/) (2023)
- [Unsupervised Layer-wise Score Aggregation for Textual OOD Detection](https://mlsanity.com/paper/2302.09852/) (2023)
- [A Benchmark for Understanding Dialogue Safety in Mental Health Support](https://mlsanity.com/paper/2307.16457/) (2023)
- [Through the Lens of Core Competency: Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2308.07902/) (2023)
- [Language Detoxification with Attribute-Discriminative Latent Space](https://mlsanity.com/paper/2210.10329/) (2022)
- [PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model](https://mlsanity.com/paper/2203.17090/) (2022)
- [Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation](https://mlsanity.com/paper/2205.09314/) (2022)
- [Stanceosaurus: Classifying Stance Towards Multilingual Misinformation](https://mlsanity.com/paper/2210.15954/) (2022)
- [Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge](https://mlsanity.com/paper/2407.03958/) (2024)
- [Improved Instruction Ordering in Recipe-Grounded Conversation](https://mlsanity.com/paper/2305.17280/) (2023)
- [A Block Metropolis-Hastings Sampler for Controllable Energy-based Text Generation](https://mlsanity.com/paper/2312.04510/) (2023)
- [The Unseen Targets of Hate -- A Systematic Review of Hateful Communication Datasets](https://mlsanity.com/paper/2405.08562/) (2024)

## Abstract (arXiv)

Dialogue models trained on human conversations inadvertently learn to generate toxic responses. In addition to producing explicitly offensive utterances, these models can also implicitly insult a group or individual by aligning themselves with an offensive statement. To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations. Specifically, we create ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model responses labeled with offensive language and stance. Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments. This undesirable behavior is learned by neural dialogue models, such as DialoGPT, which we show are two times more likely to agree with offensive comments. To enable automatic detection of offensive language, we fine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for offensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the effectiveness of controllable text generation (CTG) methods to mitigate the tendency of neural dialogue models to agree with offensive comments. Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and produces 29% fewer offensive replies. Our work highlights the need for further efforts to characterize and analyze inappropriate behavior in dialogue models, in order to help make them safer. Our code and corpus are available at https://github.com/abaheti95/ToxiChat .