mlsanityData to 7 Oct 2026

On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark

arXiv 2110.08466 ↗First posted 16 Oct 2021cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

Dialogue safety problems severely limit the real-world deployment of neural conversational models and have attracted great research interests recently. However, dialogue safety problems remain under-defined and the corresponding dataset is scarce. We propose a taxonomy for dialogue safety specifically designed to capture unsafe behaviors in human-bot dialogue settings, with focuses on context-sensitive unsafety, which is under-explored in prior works. To spur research in this direction, we compile DiaSafety, a dataset with rich context-sensitive unsafe examples. Experiments show that existing safety guarding tools fail severely on our dataset. As a remedy, we train a dialogue safety classifier to provide a strong baseline for context-sensitive dialogue unsafety detection. With our classifier, we perform safety evaluations on popular conversational models and show that existing dialogue systems still exhibit concerning context-sensitive safety problems.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 41 of its 83 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

41 of 83 references.

References · 41

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  7. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  8. 2020Recipes for building an open-domain chatbotcited by 593
  9. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  10. 2018Gender Bias in Coreference Resolutioncited by 438
  11. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  12. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
29 more
  1. 2020The Pushshift Reddit Datasetcited by 289
  2. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  3. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  4. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  5. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182
  6. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  7. 2018Learning Gender-Neutral Word Embeddingscited by 169
  8. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  9. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  10. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  11. 2017ParlAI: A Dialog Research Software Platformcited by 139
  12. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  13. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  14. 2021PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Supportcited by 92
  15. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  16. 2018From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbotscited by 85
  17. 2019Dreaddit: A Reddit Dataset for Stress Analysis in Social Mediacited by 81
  18. 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
  19. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  20. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  21. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  22. 2017Detecting Online Hate Speech Using Context Aware Modelscited by 72
  23. 2018Thou shalt not hate: Countering Online Hate Speechcited by 64
  24. 2021Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contextscited by 56
  25. 2020Controlling Style in Generated Dialoguecited by 38
  26. 2021A systematic review of Hate Speech automatic detection using Natural Language Processingcited by 26
  27. 2019TalkDown: A Corpus for Condescension Detection in Contextcited by 21
  28. 2021Assessing Political Prudence of Open-domain Chatbotscited by 20
  29. 2021Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detectioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 20; 2024: 21; 2025: 10; 2026: 4.

73 papers.

Sort:

By area:

  1. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  2. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  3. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 90
  4. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  5. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
  6. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  7. 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 74
  8. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  9. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 52
  10. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 48
  11. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  12. 2024Evaluating Cultural and Social Awareness of LLM Web Agentscited by 43
38 more
  1. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
  2. 2023Using In-Context Learning to Improve Dialogue Safetycited by 41
  3. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  4. 2022Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarkscited by 36
  5. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  6. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  7. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  8. 2023Mirages: On Anthropomorphism in Dialogue Systemscited by 30
  9. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  10. 2023KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applicationcited by 24
  11. 2025Steering MoE LLMs via Expert (De)Activationcited by 23
  12. 2023FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicitycited by 22
  13. 2022MoralDial: A Framework to Train and Evaluate Moral Dialogue Systems via Moral Discussionscited by 17
  14. 2023Bring Your Own Data! Self-Supervised Evaluation for Large Language Modelscited by 17
  15. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14
  16. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  17. 2023A Benchmark for Understanding Dialogue Safety in Mental Health Supportcited by 13
  18. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Modelscited by 12
  19. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Modelscited by 12
  20. 2022Risk-graded Safety for Handling Medical Queries in Conversational AIcited by 11
  21. 2022Language Detoxification with Attribute-Discriminative Latent Spacecited by 11
  22. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  23. 2024Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarioscited by 10
  24. 2024SafeWorld: Geo-Diverse Safety Alignmentcited by 10
  25. 2022PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Modelcited by 9
  26. 2022Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systemscited by 9
  27. 2025SafeLawBench: Towards Safe Alignment of Large Language Modelscited by 9
  28. 2022Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentationcited by 8
  29. 2022Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generationcited by 8
  30. 2024NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalismcited by 8
  31. 2022Towards Robust NLG Bias Evaluation with Syntactically-diverse Promptscited by 7
  32. 2024Coherence-Driven Multimodal Safety Dialogue with Active Learning for Embodied Agentscited by 7
  33. 2024Building a Domain-specific Guardrail Model in Productioncited by 6
  34. 2024Conversational Complexity for Assessing Risk in Large Language Modelscited by 6
  35. 2022Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AIcited by 5
  36. 2024Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspectivecited by 5
  37. 2024"Yes, My LoRD." Guiding Language Model Extraction with Locality Reinforced Distillationcited by 5
  38. 2024Responsible Retrieval Augmented Generation for Climate Decision Making from Documentscited by 4

50 most cited of 73 All 73 (JSON)