mlsanityData to 14 Oct 2026

ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation

arXiv 2310.17389 ↗First posted 26 Oct 2023cs.CL cs.AIcited by 231

Not on the map. Why?

Cited by 107 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite remarkable advances that large language models have achieved in chatbots, maintaining a non-toxic user-AI interactive environment has become increasingly critical nowadays. However, previous efforts in toxicity detection have been mostly based on benchmarks derived from social media content, leaving the unique challenges inherent to real-world user-AI interactions insufficiently explored. In this work, we introduce ToxicChat, a novel benchmark based on real user queries from an open-source chatbot. This benchmark contains the rich, nuanced phenomena that can be tricky for current toxicity detection models to identify, revealing a significant domain difference compared to social media content. Our systematic evaluation of models trained on existing toxicity datasets has shown their shortcomings when applied to this unique domain of ToxicChat. Our work illuminates the potentially overlooked challenges of toxicity detection in real-world user-AI conversations. In the future, ToxicChat can be a valuable resource to drive further advancements toward building a safe and healthy environment for user-AI interactions.

Key facts. Cited by 231 papers in the map (data through 14 Oct 2026). Builds on 11 of its 26 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

11 of 26 references.

References · 11

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  4. 2016FastText.zip: Compressing text classification modelscited by 335
  5. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 207
  6. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  7. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  8. 2021ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AIcited by 46
  9. 2021Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimationcited by 18
  10. 2019A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agentscited by 12
  11. 2020Detect All Abuse! Toward Universal Abusive Language Detection Modelscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 60; 2025: 74; 2026: 95.

231 papers.

Sort:

By area:

  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  3. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  4. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  5. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  7. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  8. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  9. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  10. 2025Qwen3Guard Technical Reportcited by 155
  11. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  12. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
38 more
  1. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  4. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 100
  5. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  6. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 96
  7. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  8. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  9. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  10. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  11. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 76
  12. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
  13. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 68
  14. 2025Safety in Large Reasoning Models: A Surveycited by 64
  15. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
  16. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 43
  17. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 42
  18. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 39
  19. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!cited by 37
  20. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 34
  21. 2024A Survey of Useful LLM Evaluationcited by 29
  22. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 29
  23. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 28
  24. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  25. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  26. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  27. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  28. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 23
  29. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 23
  30. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23
  31. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 22
  32. 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 21
  33. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 20
  34. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 20
  35. 2024HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Modelscited by 19
  36. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 19
  37. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 19
  38. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18

50 most cited of 231 All 231 (JSON)