mlsanityData to 7 Oct 2026

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling

arXiv 2107.03451 ↗First posted 7 Jul 2021cs.CL cs.AIcited by 77

Not on the map. Why?

Abstract (arXiv)

Over the last several years, end-to-end neural conversational agents have vastly improved in their ability to carry a chit-chat conversation with humans. However, these models are often trained on large datasets from the internet, and as a result, may learn undesirable behaviors from this data, such as toxic or otherwise harmful language. Researchers must thus wrestle with the issue of how and when to release these models. In this paper, we survey the problem landscape for safety for end-to-end conversational AI and discuss recent and related work. We highlight tensions between values, potential positive impact and potential harms, and provide a framework for making decisions about whether and how to release these models, following the tenets of value-sensitive design. We additionally provide a suite of tools to enable researchers to make better-informed decisions about training and releasing end-to-end conversational AI models.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 87 of its 224 references that are in the map. First posted on arXiv 7 Jul 2021.

Builds on

87 of 224 references.

References · 87

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2018Model Cards for Model Reportingcited by 1,068
  11. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
38 more
  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  2. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  5. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  6. 2020Recipes for building an open-domain chatbotcited by 593
  7. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  8. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  9. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  10. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  13. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  14. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  15. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  16. 2020The Pushshift Reddit Datasetcited by 289
  17. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  18. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  19. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  20. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  21. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  22. 2015A Neural Conversational Modelcited by 195
  23. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  24. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  25. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  26. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  27. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  28. 2018Neural Approaches to Conversational AIcited by 164
  29. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  30. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  31. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  32. 2020SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)cited by 138
  33. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  34. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  35. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  36. 2020A Distributional Approach to Controlled Text Generationcited by 117
  37. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  38. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 31; 2023: 21; 2024: 12; 2025: 5.

77 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  7. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  8. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  9. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  10. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  11. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  12. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
38 more
  1. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 92
  2. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  3. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  4. 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 74
  5. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  6. 2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
  7. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  8. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  9. 2021ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AIcited by 46
  10. 2023Using In-Context Learning to Improve Dialogue Safetycited by 41
  11. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  12. 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
  13. 2021SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failurescited by 37
  14. 2022Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarkscited by 36
  15. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  16. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  17. 2021A Word on Machine Ethics: A Response to Jiang et al. (2021)cited by 23
  18. 2022Robust Conversational Agents against Imperceptible Toxicity Triggerscited by 23
  19. 2022Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Postscited by 23
  20. 2023CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Modelscited by 22
  21. 2022Cultural Incongruencies in Artificial Intelligencecited by 21
  22. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  23. 2021Refocusing on Relevance: Personalization in NLGcited by 18
  24. 2022MoralDial: A Framework to Train and Evaluate Moral Dialogue Systems via Moral Discussionscited by 17
  25. 2024Shortcomings of LLMs for Low-Resource Translation: Retrieval and Understanding are Both the Problemcited by 17
  26. 2022MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 15
  27. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14
  28. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  29. 2022Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trollscited by 11
  30. 2022Towards Boosting the Open-Domain Chatbot with Human Feedbackcited by 11
  31. 2022Risk-graded Safety for Handling Medical Queries in Conversational AIcited by 11
  32. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
  33. 2023Improving Open Language Models by Learning from Organic Interactionscited by 10
  34. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 10
  35. 2024RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Processcited by 10
  36. 2024Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarioscited by 10
  37. 2022PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Modelcited by 9
  38. 2022Computer says "No": The Case Against Empathetic Conversational AIcited by 9

50 most cited of 77 All 77 (JSON)