# Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators

arXiv: https://arxiv.org/abs/2411.05854 · HTML: https://mlsanity.com/paper/2411.05854/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 33 of its 88 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 33)

- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [The science of fake news](https://mlsanity.com/paper/2307.07903/) (2023)
- [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021)
- [Challenges for Toxic Comment Classification: An In-Depth Error Analysis](https://mlsanity.com/paper/1809.07572/) (2018)
- [Auditing Radicalization Pathways on YouTube](https://mlsanity.com/paper/1908.08313/) (2019)
- [Targeting the Benchmark: On Methodology in Current Natural Language Processing Research](https://mlsanity.com/paper/2007.04792/) (2020)
- [Stop Clickbait: Detecting and Preventing Clickbaits in Online News Media](https://mlsanity.com/paper/1610.09786/) (2016)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [HowToCaption: Prompting LLMs to Transform Video Annotations at Scale](https://mlsanity.com/paper/2310.04900/) (2023)
- [Detecting Harmful Content On Online Platforms: What Platforms Need Vs. Where Research Efforts Go](https://mlsanity.com/paper/2103.00153/) (2021)
- [A Framework of Severity for Harmful Content Online](https://mlsanity.com/paper/2108.04401/) (2021)
- [Towards Robust Toxic Content Classification](https://mlsanity.com/paper/1912.06872/) (2019)
- [Prompt Stability Scoring for Text Annotation with Large Language Models](https://mlsanity.com/paper/2407.02039/) (2024)
- [Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization](https://mlsanity.com/paper/1912.11211/) (2019)
- [YouNICon: YouTube's CommuNIty of Conspiracy Videos](https://mlsanity.com/paper/2304.05274/) (2023)
- ["Community Guidelines Make this the Best Party on the Internet": An In-Depth Study of Online Platforms' Content Moderation Policies](https://mlsanity.com/paper/2405.05225/) (2024)
- [But Who Protects the Moderators? The Case of Crowdsourced Image Moderation](https://mlsanity.com/paper/1804.10999/) (2018)
- [Are Chatbots Reliable Text Annotators? Sometimes](https://mlsanity.com/paper/2311.05769/) (2023)
- [A Machine Learning Approach to Comment Toxicity Classification](https://mlsanity.com/paper/1903.06765/) (2019)
- [Auditing YouTube's Recommendation Algorithm for Misinformation Filter Bubbles](https://mlsanity.com/paper/2210.10085/) (2022)
- [Are Anti-Feminist Communities Gateways to the Far Right? Evidence from Reddit and YouTube](https://mlsanity.com/paper/2102.12837/) (2021)
- [Accuracy of a Large Language Model in Distinguishing Anti- And Pro-vaccination Messages on Social Media: The Case of Human Papillomavirus Vaccination](https://mlsanity.com/paper/2404.06731/) (2024)

## Built on by (most cited first, 8 of 8)

- [VLM as Policy: Common-Law Content Moderation Framework for Short Video Platform](https://mlsanity.com/paper/2504.14904/) (2025)
- [Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling](https://mlsanity.com/paper/2508.03296/) (2025)
- [Re-ranking Using Large Language Models for Mitigating Exposure to Harmful Content on Social Media Platforms](https://mlsanity.com/paper/2501.13977/) (2025)
- [Evaluating AI capabilities in detecting conspiracy theories on YouTube](https://mlsanity.com/paper/2505.23570/) (2025)
- [MetaHarm: Harmful YouTube Video Dataset Annotated by Domain Experts, GPT-4-Turbo, and Crowdworkers](https://mlsanity.com/paper/2504.16304/) (2025)
- [When Youth Enter the Algorithmic Wild: Discovering and Understanding Potentially Harmful Teen Videos on Douyin and Kwai](https://mlsanity.com/paper/2605.23598/) (2026)
- [Child-Oriented AIGC Video Risk Reviewing: A Benchmark and Knowledge-Supported Iterative Reasoning Framework](https://mlsanity.com/paper/2607.22715/) (2026)
- [Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study](https://mlsanity.com/paper/2608.17583/) (2026)

## Abstract (arXiv)

Short video platforms, such as YouTube, Instagram, or TikTok, are used by billions of users globally. These platforms expose users to harmful content, ranging from clickbait or physical harms to misinformation or online hate. Yet, detecting harmful videos remains challenging due to an inconsistent understanding of what constitutes harm and limited resources and mental tolls involved in human annotation. As such, this study advances measures and methods to detect harm in video content. First, we develop a comprehensive taxonomy for online harm on video platforms, categorizing it into six categories: Information, Hate and harassment, Addictive, Clickbait, Sexual, and Physical harms. Next, we establish multimodal large language models as reliable annotators of harmful videos. We analyze 19,422 YouTube videos using 14 image frames, 1 thumbnail, and text metadata, comparing the accuracy of crowdworkers (Mturk) and GPT-4-Turbo with domain expert annotations serving as the gold standard. Our results demonstrate that GPT-4-Turbo outperforms crowdworkers in both binary classification (harmful vs. harmless) and multi-label harm categorization tasks. Methodologically, this study extends the application of LLMs to multi-label and multi-modal contexts beyond text annotation and binary classification. Practically, our study contributes to online harm mitigation by guiding the definitions and identification of harmful content on video platforms.