# Harmful Suicide Content Detection

arXiv: https://arxiv.org/abs/2407.13942 · HTML: https://mlsanity.com/paper/2407.13942/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Depression and Self-Harm Risk Assessment in Online Forums](https://mlsanity.com/paper/1709.01848/) (2017)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)
- [Less is More for Long Document Summary Evaluation by LLMs](https://mlsanity.com/paper/2309.07382/) (2023)
- [ScAN: Suicide Attempt and Ideation Events Dataset](https://mlsanity.com/paper/2205.07872/) (2022)
- [Weakly-Supervised Methods for Suicide Risk Assessment: Role of Related Domains](https://mlsanity.com/paper/2106.02792/) (2021)
- [Analyzing Norm Violations in Live-Stream Chat](https://mlsanity.com/paper/2305.10731/) (2023)

## Abstract (arXiv)

Harmful suicide content on the Internet is a significant risk factor inducing suicidal thoughts and behaviors among vulnerable populations. Despite global efforts, existing resources are insufficient, specifically in high-risk regions like the Republic of Korea. Current research mainly focuses on understanding negative effects of such content or suicide risk in individuals, rather than on automatically detecting the harmfulness of content. To fill this gap, we introduce a harmful suicide content detection task for classifying online suicide content into five harmfulness levels. We develop a multi-modal benchmark and a task description document in collaboration with medical professionals, and leverage large language models (LLMs) to explore efficient methods for moderating such content. Our contributions include proposing a novel detection task, a multi-modal Korean benchmark with expert annotations, and suggesting strategies using LLMs to detect illegal and harmful content. Owing to the potential harm involved, we publicize our implementations and benchmark, incorporating an ethical verification process.