# SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures

arXiv: https://arxiv.org/abs/2110.07518 · HTML: https://mlsanity.com/paper/2110.07518/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://mlsanity.com/paper/2004.08449/) (2020)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech](https://mlsanity.com/paper/2106.01625/) (2021)
- [Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions](https://mlsanity.com/paper/2006.12442/) (2020)
- [Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations](https://mlsanity.com/paper/2008.12348/) (2020)
- [A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agents](https://mlsanity.com/paper/1909.04387/) (2019)
- [Countering Online Hate Speech: An NLP Perspective](https://mlsanity.com/paper/2109.02941/) (2021)

## Built on by (most cited first, 30 of 37)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey](https://mlsanity.com/paper/2402.09283/) (2024)
- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference](https://mlsanity.com/paper/2406.17626/) (2024)
- [Using In-Context Learning to Improve Dialogue Safety](https://mlsanity.com/paper/2302.00871/) (2023)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks](https://mlsanity.com/paper/2202.08011/) (2022)
- [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022)
- [Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models](https://mlsanity.com/paper/2305.17878/) (2023)
- [Safer-Instruct: Aligning Language Models with Automated Preference Data](https://mlsanity.com/paper/2311.08685/) (2023)
- [NLP for Counterspeech against Hate: A Survey and How-To Guide](https://mlsanity.com/paper/2403.20103/) (2024)
- [Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation](https://mlsanity.com/paper/2310.14424/) (2023)
- [Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?](https://mlsanity.com/paper/2310.15758/) (2023)
- [Mitigating Social Biases in Language Models through Unlearning](https://mlsanity.com/paper/2406.13551/) (2024)
- [Purple-teaming LLMs with Adversarial Defender Training](https://mlsanity.com/paper/2407.01850/) (2024)
- [Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection](https://mlsanity.com/paper/2510.13893/) (2025)
- [VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems](https://mlsanity.com/paper/2605.17467/) (2026)
- [DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines](https://mlsanity.com/paper/2212.10557/) (2022)
- [Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning](https://mlsanity.com/paper/2410.12621/) (2024)
- [Effortless Integration of Memory Management into Open-Domain Conversation Systems](https://mlsanity.com/paper/2305.13973/) (2023)
- [CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs](https://mlsanity.com/paper/2311.17376/) (2023)
- [Learning from Implicit User Feedback, Emotions and Demographic Information in Task-Oriented and Document-Grounded Dialogues](https://mlsanity.com/paper/2401.09248/) (2024)
- [The Life Cycle of Large Language Models: A Review of Biases in Education](https://mlsanity.com/paper/2407.11203/) (2024)
- [Chained Tuning Leads to Biased Forgetting](https://mlsanity.com/paper/2412.16469/) (2024)
- [Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems](https://mlsanity.com/paper/2211.10596/) (2022)
- [Sources of Noise in Dialogue and How to Deal with Them](https://mlsanity.com/paper/2212.02745/) (2022)

## Abstract (arXiv)

Current open-domain conversational models can easily be made to talk in inadequate ways. Online learning from conversational feedback given by the conversation partner is a promising avenue for a model to improve and adapt, so as to generate fewer of these safety failures. However, current state-of-the-art models tend to react to feedback with defensive or oblivious responses. This makes for an unpleasant experience and may discourage conversation partners from giving feedback in the future. This work proposes SaFeRDialogues, a task and dataset of graceful responses to conversational feedback about safety failures. We collect a dataset of 10k dialogues demonstrating safety failures, feedback signaling them, and a response acknowledging the feedback. We show how fine-tuning on this dataset results in conversations that human raters deem considerably more likely to lead to a civil conversation, without sacrificing engagingness or general conversational ability.