mlsanityData to 7 Oct 2026

"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media

arXiv 2304.10619 ↗First posted 20 Apr 2023cs.CL cs.AI cs.HCcited by 45

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Harmful content is pervasive on social media, poisoning online communities and negatively impacting participation. A common approach to address this issue is to develop detection models that rely on human annotations. However, the tasks required to build such models expose annotators to harmful and offensive content and may require significant time and cost to complete. Generative AI models have the potential to understand and detect harmful content. To investigate this potential, we used ChatGPT and compared its performance with MTurker annotations for three frequently discussed concepts related to harmful content: Hateful, Offensive, and Toxic (HOT). We designed five prompts to interact with ChatGPT and conducted four experiments eliciting HOT classifications. Our results show that ChatGPT can achieve an accuracy of approximately 80% when compared to MTurker annotations. Specifically, the model displays a more consistent classification for non-HOT comments than HOT comments compared to human annotations. Our findings also suggest that ChatGPT classifications align with provided HOT definitions, but ChatGPT classifies "hateful" and "offensive" as subsets of "toxic." Moreover, the choice of prompts used to interact with ChatGPT impacts its performance. Based on these in-sights, our study provides several meaningful implications for employing ChatGPT to detect HOT content, particularly regarding the reliability and consistency of its performance, its understand-ing and reasoning of the HOT concept, and the impact of prompts on its performance. Overall, our study provides guidance about the potential of using generative AI models to moderate large volumes of user-generated content on social media.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 31 of its 107 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

31 of 107 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  6. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  9. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
  10. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  11. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  12. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
19 more
  1. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  2. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  3. 2017Deep Learning for Hate Speech Detection in Tweetscited by 180
  4. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  5. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  6. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  7. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  8. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  9. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 56
  10. 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
  11. 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 51
  12. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  13. 2017Detecting Hate Speech in Social Mediacited by 42
  14. 2022Toxicity Detection with Generative Prompt-based Inferencecited by 35
  15. 2023Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferencescited by 25
  16. 2022L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT modelscited by 20
  17. 2023Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defensecited by 13
  18. 2019Detecting Cyberbullying and Cyberaggression in Social Mediacited by 11
  19. 2022Remove, Reduce, Inform: What Actions do People Want Social Media Platforms to Take on Potentially Misleading Content?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 17; 2025: 17; 2026: 3.

45 papers.

Sort:

By area:

  1. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  2. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  3. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  4. 2024Content Moderation by LLM: From Accuracy to Legitimacycited by 30
  5. 2023Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuningcited by 28
  6. 2024Don't Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detectioncited by 25
  7. 2024Personas with Attitudes: Controlling LLMs for Diverse Data Annotationcited by 21
  8. 2024An Investigation of Large Language Models for Real-World Hate Speech Detectioncited by 20
  9. 2023Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Modelscited by 17
  10. 2024Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationalescited by 16
  11. 2023Watch Your Language: Investigating Content Moderation with Large Language Modelscited by 14
  12. 2024Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Wayscited by 14
33 more
  1. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labelingcited by 8
  2. 2024Emojis Decoded: Leveraging ChatGPT for Enhanced Understanding in Social Media Communicationscited by 7
  3. 2025Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Studycited by 7
  4. 2023Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detectioncited by 6
  5. 2025Evaluating how LLM annotations represent diverse views on contentious topicscited by 6
  6. 2025A Modular Taxonomy for Hate Speech Definitions and Its Impact on Zero-Shot LLM Classification Performancecited by 6
  7. 2025Generative Data Refinement: Just Ask for Better Datacited by 6
  8. 2025From Promising Capability to Pervasive Bias: Assessing Large Language Models for Emergency Department Triagecited by 5
  9. 2024Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Modelscited by 4
  10. 2024RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detectioncited by 4
  11. 2025Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognitioncited by 4
  12. 2024OffensiveLang: A Community Based Implicit Offensive Language Datasetcited by 2
  13. 2024The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AIcited by 2
  14. 2024Leveraging Large Language Models and Topic Modeling for Toxicity Classificationcited by 2
  15. 2025Digital Guardians: Can GPT-4, Perspective API, and Moderation API reliably detect hate speech in reader comments of German online newspapers?cited by 2
  16. 2025Improving Harmful Text Detection with Joint Retrieval and External Knowledgecited by 2
  17. 2024Towards Generalized Offensive Language Identificationcited by 1
  18. 2024Hate Speech Detection Using Cross-Platform Social Media Data In English and German Languagecited by 1
  19. 2025Detecting harassment and defamation in cyberbullying with emotion-adaptive trainingcited by 1
  20. 2025Specializing General-purpose LLM Embeddings for Implicit Hate Speech Detection across Datasetscited by 1
  21. 2025Language, Culture, and Ideology: Personalizing Offensiveness Detection in Political Tweets with Reasoning LLMscited by 1
  22. 2023A Survey on Online User Aggression: Content Detection and Behavioral Analysis on Social Mediacited by 0
  23. 2024Unveiling Social Media Comments with a Novel Named Entity Recognition System for Identity Groupscited by 0
  24. 2024Safe Guard: an LLM-agent for Real-time Voice-based Hate Speech Detection in Social Virtual Realitycited by 0
  25. 2024"Is Hate Lost in Translation?": Evaluation of Multilingual LGBTQIA+ Hate Speech Detectioncited by 0
  26. 2025OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Modelscited by 0
  27. 2025Recalibrating the Compass: Integrating Large Language Models into Classical Research Methodscited by 0
  28. 2025Context-Aware Content Moderation for German Newspaper Commentscited by 0
  29. 2025The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderationcited by 0
  30. 2025Defining, Understanding, and Detecting Online Toxicity: Challenges and Machine Learning Approachescited by 0
  31. 2026Robust Persona-Aware Toxicity Detection with Prompt Optimization and Learned Ensemblingcited by 0
  32. 2026You Frame It: How Conceptual Representations Shape LLM Detection and Reasoning about Antisemitismcited by 0
  33. 2026From Specialization to Generalization: Instruction-tuned LLMs for Robust Harmful Content Mitigationcited by 0