# Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study

arXiv: https://arxiv.org/abs/2505.21710 · HTML: https://mlsanity.com/paper/2505.21710/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 44 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 7 of 7)

- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Do Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incels](https://mlsanity.com/paper/2010.10397/) (2020)
- [Proactive Moderation of Online Discussions: Existing Practices and the Potential for Algorithmic Support](https://mlsanity.com/paper/2211.16525/) (2022)
- [Watch Your Language: Investigating Content Moderation with Large Language Models](https://mlsanity.com/paper/2309.14517/) (2023)
- [Exploring ChatGPT and its Impact on Society](https://mlsanity.com/paper/2403.14643/) (2024)
- [Cross-Domain Toxic Spans Detection](https://mlsanity.com/paper/2306.09642/) (2023)

## Abstract (arXiv)

This study evaluates the effectiveness of ChatGPT, an advanced AI model for natural language processing, in identifying targeting and inappropriate language in online comments. With the increasing challenge of moderating vast volumes of user-generated content on social network sites, the role of AI in content moderation has gained prominence. We compared ChatGPT's performance against crowd-sourced annotations and expert evaluations to assess its accuracy, scope of detection, and consistency. Our findings show that ChatGPT performs well in detecting inappropriate content, and that almost the entire gain from iterative prompt refinement came from a single change --- simplifying the label scheme and moving to GPT-4 --- after which further adjustments to prompt structure and sampling temperature had little effect. Its performance on targeting language was weaker and less stable, with higher false positive rates than expert judgments before refinement and higher false negative rates after a change of model. This study contributes to the field by demonstrating the potential of AI models like ChatGPT to enhance automated content moderation systems while also identifying areas for further improvement. The results underscore the importance of careful prompt design and model selection, and of contextual understanding, in supporting automated moderation and mitigating harmful online behavior.