# Advancing Content Moderation: Evaluating Large Language Models for Detecting Sensitive Content Across Text, Images, and Videos

arXiv: https://arxiv.org/abs/2411.17123 · HTML: https://mlsanity.com/paper/2411.17123/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 118 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [Comparative Study of CNN and RNN for Natural Language Processing](https://mlsanity.com/paper/1702.01923/) (2017)
- [Hate Speech Detection from Code-mixed Hindi-English Tweets Using Deep Learning Models](https://mlsanity.com/paper/1811.05145/) (2018)
- [Fine-Tuning Llama 2 Large Language Models for Detecting Online Sexual Predatory Chats and Abusive Texts](https://mlsanity.com/paper/2308.14683/) (2023)
- [Generating Educational Materials with Different Levels of Readability using LLMs](https://mlsanity.com/paper/2406.12787/) (2024)
- [Applying deep learning to classify pornographic images and videos](https://mlsanity.com/paper/1511.08899/) (2015)
- [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024)
- [HASOCOne@FIRE-HASOC2020: Using BERT and Multilingual BERT models for Hate Speech Detection](https://mlsanity.com/paper/2101.09007/) (2021)
- [Machine Learning to study the impact of gender-based violence in the news media](https://mlsanity.com/paper/2012.07490/) (2020)
- [Real-Time Violence Detection Using CNN-LSTM](https://mlsanity.com/paper/2107.07578/) (2021)

## Built on by (most cited first, 10 of 10)

- [Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models](https://mlsanity.com/paper/2502.18695/) (2025)
- [Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching](https://mlsanity.com/paper/2512.03553/) (2025)
- [VModA: An Effective Framework for Adaptive NSFW Image Moderation](https://mlsanity.com/paper/2505.23386/) (2025)
- [CompAgent: An Agentic Framework for Visual Compliance Verification](https://mlsanity.com/paper/2511.00171/) (2025)
- [From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation](https://mlsanity.com/paper/2606.30059/) (2026)
- [Context-Aware Toxicity Detection in Multiplayer Games: Integrating Domain-Adaptive Pretraining and Match Metadata](https://mlsanity.com/paper/2504.01534/) (2025)
- [Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models](https://mlsanity.com/paper/2508.15810/) (2025)
- [Why Do Large Language Models Generate Harmful Content?](https://mlsanity.com/paper/2604.11663/) (2026)
- [What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks](https://mlsanity.com/paper/2606.09700/) (2026)
- [VAMR: Multi-Question Agentic Reasoning for Efficient Long-Form Video Understanding](https://mlsanity.com/paper/2610.11171/) (2026)

## Abstract (arXiv)

The widespread dissemination of hate speech, harassment, harmful and sexual content, and violence across websites and media platforms presents substantial challenges and provokes widespread concern among different sectors of society. Governments, educators, and parents are often at odds with media platforms about how to regulate, control, and limit the spread of such content. Technologies for detecting and censoring the media contents are a key solution to addressing these challenges. Techniques from natural language processing and computer vision have been used widely to automatically identify and filter out sensitive content such as offensive languages, violence, nudity, and addiction in both text, images, and videos, enabling platforms to enforce content policies at scale. However, existing methods still have limitations in achieving high detection accuracy with fewer false positives and false negatives. Therefore, more sophisticated algorithms for understanding the context of both text and image may open rooms for improvement in content censorship to build a more efficient censorship system. In this paper, we evaluate existing LLM-based content moderation solutions such as OpenAI moderation model and Llama-Guard3 and study their capabilities to detect sensitive contents. Additionally, we explore recent LLMs such as GPT, Gemini, and Llama in identifying inappropriate contents across media outlets. Various textual and visual datasets like X tweets, Amazon reviews, news articles, human photos, cartoons, sketches, and violence videos have been utilized for evaluation and comparison. The results demonstrate that LLMs outperform traditional techniques by achieving higher accuracy and lower false positive and false negative rates. This highlights the potential to integrate LLMs into websites, social media platforms, and video-sharing services for regulatory and content moderation purposes.