# Sensitive Content Classification in Social Media: A Holistic Resource and Evaluation

arXiv: https://arxiv.org/abs/2411.19832 · HTML: https://mlsanity.com/paper/2411.19832/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 71 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy](https://mlsanity.com/paper/2307.00691/) (2023)
- [Directions in Abusive Language Training Data: Garbage In, Garbage Out](https://mlsanity.com/paper/2004.01670/) (2020)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://mlsanity.com/paper/2404.12241/) (2024)
- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [Detecting Harmful Content On Online Platforms: What Platforms Need Vs. Where Research Efforts Go](https://mlsanity.com/paper/2103.00153/) (2021)
- [A Framework of Severity for Harmful Content Online](https://mlsanity.com/paper/2108.04401/) (2021)
- [Security and Privacy for Artificial Intelligence: Opportunities and Challenges](https://mlsanity.com/paper/2102.04661/) (2021)

## Built on by (most cited first, 4 of 4)

- [CRADLE Bench: A Clinician-Annotated Benchmark for Multi-Faceted Mental Health Crisis and Safety Risk Detection](https://mlsanity.com/paper/2510.23845/) (2025)
- [When Harmful Content Gets Camouflaged: Unveiling Perception Failure of LVLMs with CamHarmTI](https://mlsanity.com/paper/2512.03087/) (2025)
- [Analysing Self-Harm Representations in Language Models: a Cross-Architecture Study](https://mlsanity.com/paper/2607.21988/) (2026)
- [Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation](https://mlsanity.com/paper/2609.03814/) (2026)

## Abstract (arXiv)

The detection of sensitive content in large datasets is crucial for ensuring that shared and analysed data is free from harmful material. However, current moderation tools, such as external APIs, suffer from limitations in customisation, accuracy across diverse sensitive categories, and privacy concerns. Additionally, existing datasets and open-source models focus predominantly on toxic language, leaving gaps in detecting other sensitive categories such as substance abuse or self-harm. In this paper, we put forward a unified dataset tailored for social media content moderation across six sensitive categories: conflictual language, profanity, sexually explicit material, drug-related content, self-harm, and spam. By collecting and annotating data with consistent retrieval strategies and guidelines, we address the shortcomings of previous focalised research. Our analysis demonstrates that fine-tuning large language models (LLMs) on this novel dataset yields significant improvements in detection performance compared to open off-the-shelf models such as LLaMA, and even proprietary OpenAI models, which underperform by 10-15% overall. This limitation is even more pronounced on popular moderation APIs, which cannot be easily tailored to specific sensitive content categories, among others.