# Safer Conversational AI as a Source of User Delight

arXiv: https://arxiv.org/abs/2304.09865 · HTML: https://mlsanity.com/paper/2304.09865/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 27 of 27)

- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning](https://mlsanity.com/paper/2006.16779/) (2020)
- [Conversational AI: The Science Behind the Alexa Prize](https://mlsanity.com/paper/1801.03604/) (2018)
- [Polite Dialogue Generation Without Parallel Data](https://mlsanity.com/paper/1805.03162/) (2018)
- [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://mlsanity.com/paper/2303.06135/) (2023)
- [Challenges for Toxic Comment Classification: An In-Depth Error Analysis](https://mlsanity.com/paper/1809.07572/) (2018)
- [Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer](https://mlsanity.com/paper/1805.07685/) (2018)
- [Controlling Style in Generated Dialogue](https://mlsanity.com/paper/2009.10855/) (2020)
- [A Unified Deep Learning Architecture for Abuse Detection](https://mlsanity.com/paper/1802.00385/) (2018)
- [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022)
- [Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations](https://mlsanity.com/paper/2008.12348/) (2020)
- [Detecting the Hate Code on Social Media](https://mlsanity.com/paper/1703.05443/) (2017)
- [A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agents](https://mlsanity.com/paper/1909.04387/) (2019)
- [Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervision](https://mlsanity.com/paper/1811.12900/) (2018)
- [Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models](https://mlsanity.com/paper/2005.13170/) (2020)
- [Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology](https://mlsanity.com/paper/2008.09706/) (2020)
- [Grounding in social media: An approach to building a chit-chat dialogue model](https://mlsanity.com/paper/2206.05696/) (2022)

## Built on by (most cited first, 2 of 2)

- [Responsible Retrieval Augmented Generation for Climate Decision Making from Documents](https://mlsanity.com/paper/2410.23902/) (2024)
- [The Chai Platform's AI Safety Framework](https://mlsanity.com/paper/2306.02979/) (2023)

## Abstract (arXiv)

This work explores the impact of moderation on users' enjoyment of conversational AI systems. While recent advancements in Large Language Models (LLMs) have led to highly capable conversational AIs that are increasingly deployed in real-world settings, there is a growing concern over AI safety and the need to moderate systems to encourage safe language and prevent harm. However, some users argue that current approaches to moderation limit the technology, compromise free expression, and limit the value delivered by the technology. This study takes an unbiased stance and shows that moderation does not necessarily detract from user enjoyment. Heavy handed moderation does seem to have a nefarious effect, but models that are moderated to be safer can lead to a better user experience. By deploying various conversational AIs in the Chai platform, the study finds that user retention can increase with a level of moderation and safe system design. These results demonstrate the importance of appropriately defining safety in models in a way that is both responsible and focused on serving users.