# A Framework of Severity for Harmful Content Online

arXiv: https://arxiv.org/abs/2108.04401 · HTML: https://mlsanity.com/paper/2108.04401/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2021.

## Built on by (most cited first, 23 of 23)

- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?](https://mlsanity.com/paper/2501.15463/) (2025)
- [VLM as Policy: Common-Law Content Moderation Framework for Short Video Platform](https://mlsanity.com/paper/2504.14904/) (2025)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies](https://mlsanity.com/paper/2408.07728/) (2024)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotators](https://mlsanity.com/paper/2411.05854/) (2024)
- [DeMod: A Holistic Tool with Explainable Detection and Personalized Modification for Toxicity Censorship](https://mlsanity.com/paper/2411.01844/) (2024)
- [Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks](https://mlsanity.com/paper/2310.16955/) (2023)
- [Supporting Human Raters with the Detection of Harmful Content using Large Language Models](https://mlsanity.com/paper/2406.12800/) (2024)
- [A Community-Centric Perspective for Characterizing and Detecting Anti-Asian Violence-Provoking Speech](https://mlsanity.com/paper/2407.15227/) (2024)
- [Sensitive Content Classification in Social Media: A Holistic Resource and Evaluation](https://mlsanity.com/paper/2411.19832/) (2024)
- [Towards Detecting Harmful Agendas in News Articles](https://mlsanity.com/paper/2302.00102/) (2023)
- [Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directions](https://mlsanity.com/paper/2509.25539/) (2025)
- [A Survey of Machine Learning Models and Datasets for the Multi-label Classification of Textual Hate Speech in English](https://mlsanity.com/paper/2504.08609/) (2025)
- [From Plausible to Causal: Counterfactual Semantics for Policy Evaluation in Simulated Online Communities](https://mlsanity.com/paper/2604.03920/) (2026)
- [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024)
- [Recourse for reclamation: Chatting with generative language models](https://mlsanity.com/paper/2403.14467/) (2024)
- [The Unappreciated Role of Intent in Algorithmic Moderation of Social Media Content](https://mlsanity.com/paper/2405.11030/) (2024)
- [MetaHarm: Harmful YouTube Video Dataset Annotated by Domain Experts, GPT-4-Turbo, and Crowdworkers](https://mlsanity.com/paper/2504.16304/) (2025)
- [Mod-Guide: An LLM-based Content Moderation Feedback System to Address Insensitive Speech toward Indigenous Ethnic and Religious Minority Communities](https://mlsanity.com/paper/2606.13397/) (2026)
- [Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech](https://mlsanity.com/paper/2607.05165/) (2026)
- [VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation](https://mlsanity.com/paper/2609.35028/) (2026)

## Abstract (arXiv)

The proliferation of harmful content on online social media platforms has necessitated empirical understandings of experiences of harm online and the development of practices for harm mitigation. Both understandings of harm and approaches to mitigating that harm, often through content moderation, have implicitly embedded frameworks of prioritization - what forms of harm should be researched, how policy on harmful content should be implemented, and how harmful content should be moderated. To aid efforts of better understanding the variety of online harms, how they relate to one another, and how to prioritize harms relevant to research, policy, and practice, we present a theoretical framework of severity for harmful online content. By employing a grounded theory approach, we developed a framework of severity based on interviews and card-sorting activities conducted with 52 participants over the course of ten months. Through our analysis, we identified four Types of Harm (physical, emotional, relational, and financial) and eight Dimensions along which the severity of harm can be understood (perspectives, intent, agency, experience, scale, urgency, vulnerability, sphere). We describe how our framework can be applied to both research and policy settings towards deeper understandings of specific forms of harm (e.g., harassment) and prioritization frameworks when implementing policies encompassing many forms of harm.