# "HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media

arXiv: https://arxiv.org/abs/2304.10619 · HTML: https://mlsanity.com/paper/2304.10619/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 31 of its 107 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels](https://mlsanity.com/paper/2203.11364/) (2022)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation](https://mlsanity.com/paper/2206.08931/) (2022)
- [A Bibliometric Review of Large Language Models Research from 2017 to 2023](https://mlsanity.com/paper/2304.02020/) (2023)
- [Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark](https://mlsanity.com/paper/2304.11085/) (2023)
- [Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation](https://mlsanity.com/paper/2112.04554/) (2021)
- [ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification](https://mlsanity.com/paper/2303.03953/) (2023)
- [Detecting Hate Speech in Social Media](https://mlsanity.com/paper/1712.06427/) (2017)
- [Toxicity Detection with Generative Prompt-based Inference](https://mlsanity.com/paper/2205.12390/) (2022)
- [Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences](https://mlsanity.com/paper/2303.07610/) (2023)
- [L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models](https://mlsanity.com/paper/2203.13778/) (2022)
- [Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense](https://mlsanity.com/paper/2303.12132/) (2023)
- [Detecting Cyberbullying and Cyberaggression in Social Media](https://mlsanity.com/paper/1907.08873/) (2019)

## Built on by (most cited first, 30 of 45)

- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes](https://mlsanity.com/paper/2312.14890/) (2023)
- [Content Moderation by LLM: From Accuracy to Legitimacy](https://mlsanity.com/paper/2409.03219/) (2024)
- [Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuning](https://mlsanity.com/paper/2310.03400/) (2023)
- [Don't Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detection](https://mlsanity.com/paper/2402.11406/) (2024)
- [Personas with Attitudes: Controlling LLMs for Diverse Data Annotation](https://mlsanity.com/paper/2410.11745/) (2024)
- [An Investigation of Large Language Models for Real-World Hate Speech Detection](https://mlsanity.com/paper/2401.03346/) (2024)
- [Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2312.15099/) (2023)
- [Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationales](https://mlsanity.com/paper/2403.12403/) (2024)
- [Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways](https://mlsanity.com/paper/2406.11980/) (2024)
- [Watch Your Language: Investigating Content Moderation with Large Language Models](https://mlsanity.com/paper/2309.14517/) (2023)
- [From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling](https://mlsanity.com/paper/2506.16393/) (2025)
- [Emojis Decoded: Leveraging ChatGPT for Enhanced Understanding in Social Media Communications](https://mlsanity.com/paper/2402.01681/) (2024)
- [Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study](https://mlsanity.com/paper/2505.06149/) (2025)
- [A Modular Taxonomy for Hate Speech Definitions and Its Impact on Zero-Shot LLM Classification Performance](https://mlsanity.com/paper/2506.18576/) (2025)
- [Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection](https://mlsanity.com/paper/2305.13276/) (2023)
- [Evaluating how LLM annotations represent diverse views on contentious topics](https://mlsanity.com/paper/2503.23243/) (2025)
- [Generative Data Refinement: Just Ask for Better Data](https://mlsanity.com/paper/2509.08653/) (2025)
- [From Promising Capability to Pervasive Bias: Assessing Large Language Models for Emergency Department Triage](https://mlsanity.com/paper/2504.16273/) (2025)
- [Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition](https://mlsanity.com/paper/2508.08274/) (2025)
- [Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Models](https://mlsanity.com/paper/2404.17985/) (2024)
- [RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2412.13008/) (2024)
- [OffensiveLang: A Community Based Implicit Offensive Language Dataset](https://mlsanity.com/paper/2403.02472/) (2024)
- [Digital Guardians: Can GPT-4, Perspective API, and Moderation API reliably detect hate speech in reader comments of German online newspapers?](https://mlsanity.com/paper/2501.01256/) (2025)
- [The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AI](https://mlsanity.com/paper/2408.05977/) (2024)
- [Leveraging Large Language Models and Topic Modeling for Toxicity Classification](https://mlsanity.com/paper/2411.17876/) (2024)
- [Improving Harmful Text Detection with Joint Retrieval and External Knowledge](https://mlsanity.com/paper/2504.02310/) (2025)
- [Towards Generalized Offensive Language Identification](https://mlsanity.com/paper/2407.18738/) (2024)
- [Hate Speech Detection Using Cross-Platform Social Media Data In English and German Language](https://mlsanity.com/paper/2410.05287/) (2024)

## Abstract (arXiv)

Harmful content is pervasive on social media, poisoning online communities and negatively impacting participation. A common approach to address this issue is to develop detection models that rely on human annotations. However, the tasks required to build such models expose annotators to harmful and offensive content and may require significant time and cost to complete. Generative AI models have the potential to understand and detect harmful content. To investigate this potential, we used ChatGPT and compared its performance with MTurker annotations for three frequently discussed concepts related to harmful content: Hateful, Offensive, and Toxic (HOT). We designed five prompts to interact with ChatGPT and conducted four experiments eliciting HOT classifications. Our results show that ChatGPT can achieve an accuracy of approximately 80% when compared to MTurker annotations. Specifically, the model displays a more consistent classification for non-HOT comments than HOT comments compared to human annotations. Our findings also suggest that ChatGPT classifications align with provided HOT definitions, but ChatGPT classifies "hateful" and "offensive" as subsets of "toxic." Moreover, the choice of prompts used to interact with ChatGPT impacts its performance. Based on these in-sights, our study provides several meaningful implications for employing ChatGPT to detect HOT content, particularly regarding the reliability and consistency of its performance, its understand-ing and reasoning of the HOT concept, and the impact of prompts on its performance. Overall, our study provides guidance about the potential of using generative AI models to moderate large volumes of user-generated content on social media.