mlsanityData to 7 Oct 2026

ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI

arXiv 2109.09483 ↗First posted 20 Sep 2021cs.CL cs.HCcited by 46

Not on the map. Why?

Abstract (arXiv)

We present the first English corpus study on abusive language towards three conversational AI systems gathered "in the wild": an open-domain social bot, a rule-based chatbot, and a task-based system. To account for the complexity of the task, we take a more `nuanced' approach where our ConvAI dataset reflects fine-grained notions of abuse, as well as views from multiple expert annotators. We find that the distribution of abuse is vastly different compared to other commonly used datasets, with more sexually tinted aggression towards the virtual persona of these systems. Finally, we report results from bench-marking existing models against this data. Unsurprisingly, we find that there is substantial room for improvement with F1 scores below 90%.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 28 of its 67 references that are in the map. First posted on arXiv 20 Sep 2021.

Builds on

28 of 67 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  3. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  4. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
  5. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  6. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 206
  7. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  8. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  9. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  10. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 139
  11. 2020SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)cited by 138
  12. 2019Multilingual and Multi-Aspect Hate Speech Analysiscited by 133
16 more
  1. 2017Understanding Abuse: A Typology of Abusive Language Detection Subtaskscited by 115
  2. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  3. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  4. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
  5. 2018All You Need is "Love": Evading Hate-speech Detectioncited by 56
  6. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 52
  7. 2020Detecting East Asian Prejudice on Social Mediacited by 40
  8. 2020The Grey Hoodie Project: Big Tobacco, Big Tech, and the threat on academic integritycited by 38
  9. 2020Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversationscited by 24
  10. 2020On Cross-Dataset Generalization in Automatic Detection of Online Abusecited by 20
  11. 2021Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detectioncited by 19
  12. 2021"Short is the Road that Leads from Fear to Hate": Fear Speech in Indian WhatsApp Groupscited by 13
  13. 2019A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agentscited by 12
  14. 2020Detect All Abuse! Toward Universal Abusive Language Detection Modelscited by 12
  15. 2018Detecting Offensive Content in Open-domain Conversations using Two Stage Semi-supervisioncited by 6
  16. 2020AbuseAnalyzer: Abuse Detection, Severity and Target Prediction for Gab Postscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 13; 2024: 13; 2025: 7; 2026: 6.

46 papers.

Sort:

By area:

  1. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  2. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  3. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  4. 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
  5. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  6. 2023Mirages: On Anthropomorphism in Dialogue Systemscited by 30
  7. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  8. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 22
  9. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  10. 2021Two Contrasting Data Annotation Paradigms for Subjective NLP Taskscited by 12
  11. 2022Risk-graded Safety for Handling Medical Queries in Conversational AIcited by 11
  12. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
34 more
  1. 2023iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?cited by 10
  2. 2024DetoxLLM: A Framework for Detoxification with Explanationscited by 10
  3. 2024Subjective textitIsms? On the Danger of Conflating Hate and Offence in Abusive Language Detectioncited by 9
  4. 2024Corpus Considerations for Annotator Modeling and Scalingcited by 8
  5. 2024Leveraging Annotator Disagreement for Text Classificationcited by 8
  6. 2025LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Taskcited by 8
  7. 2024Re-examining Sexism and Misogyny Classification with Annotator Attitudescited by 7
  8. 2024Decoding Hate: Exploring Language Models' Reactions to Hate Speechcited by 6
  9. 2023Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!cited by 5
  10. 2022The State of Profanity Obfuscation in Natural Language Processingcited by 4
  11. 2024Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groupscited by 4
  12. 2024Understanding The Effect Of Temperature On Alignment With Human Opinionscited by 4
  13. 2025Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systemscited by 4
  14. 2023From Chatter to Matter: Addressing Critical Steps of Emotion Recognition Learning in Task-oriented Dialoguecited by 3
  15. 2025The Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrumcited by 3
  16. 2026STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systemscited by 3
  17. 2022SOLD: Sinhala Offensive Language Datasetcited by 2
  18. 2023Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Taskscited by 2
  19. 2024How Many Ratings per Item are Necessary for Reliable Significance Testing?cited by 2
  20. 2022Cross-Platform and Cross-Domain Abusive Language Detection with Supervised Contrastive Learningcited by 1
  21. 2023Lon-ea at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Predictioncited by 1
  22. 2023SafeWebUH at SemEval-2023 Task 11: Learning Annotator Disagreement in Derogatory Text: Comparison of Direct Training vs Aggregationcited by 1
  23. 2024Impoverished Language Technology: The Lack of (Social) Class in NLPcited by 1
  24. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  25. 2025Subjective Logic Encodingscited by 1
  26. 2023On the Challenges of Building Datasets for Hate Speech Detectioncited by 0
  27. 2023HCDIR: End-to-end Hate Context Detection, and Intensity Reduction model for online commentscited by 0
  28. 2025Towards a comprehensive taxonomy of online abusive language informed by machine leaningcited by 0
  29. 2025Web(er) of Hate: A Survey on How Hate Speech Is Typedcited by 0
  30. 2026Understanding Annotator Safety Policy with Interpretabilitycited by 0
  31. 2026From Specialization to Generalization: Instruction-tuned LLMs for Robust Harmful Content Mitigationcited by 0
  32. 2026Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignmentcited by 0
  33. 2026How User-AI Mistreatment Occurs and Matters in Conversational Systems?cited by 0
  34. 2026How AI Assistants Respond to Repeated Abusecited by 0