# Toxicity Detection: Does Context Really Matter?

arXiv: https://arxiv.org/abs/2006.00998 · HTML: https://mlsanity.com/paper/2006.00998/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 30 of 93)

- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://mlsanity.com/paper/2310.17389/) (2023)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark](https://mlsanity.com/paper/2110.08466/) (2021)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark](https://mlsanity.com/paper/2305.14938/) (2023)
- [Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts](https://mlsanity.com/paper/2108.11830/) (2021)
- [Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications](https://mlsanity.com/paper/2311.16153/) (2023)
- [ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI](https://mlsanity.com/paper/2109.09483/) (2021)
- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [Hate Speech and Counter Speech Detection: Conversational Context Does Matter](https://mlsanity.com/paper/2206.06423/) (2022)
- [Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding](https://mlsanity.com/paper/2210.14169/) (2022)
- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [DIALKI: Knowledge Identification in Conversational Systems through Dialogue-Document Contextualization](https://mlsanity.com/paper/2109.04673/) (2021)
- [Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Models](https://mlsanity.com/paper/2307.09209/) (2023)
- [Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detection](https://mlsanity.com/paper/2103.14916/) (2021)
- [Detecting Community Sensitive Norm Violations in Online Conversations](https://mlsanity.com/paper/2110.04419/) (2021)
- [Can Authorship Representation Learning Capture Stylistic Features?](https://mlsanity.com/paper/2308.11490/) (2023)
- [ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information](https://mlsanity.com/paper/2409.14740/) (2024)
- [GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives](https://mlsanity.com/paper/2311.05074/) (2023)
- [CONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection](https://mlsanity.com/paper/2106.06213/) (2021)
- [KOLD: Korean Offensive Language Dataset](https://mlsanity.com/paper/2205.11315/) (2022)
- [Watch Your Language: Investigating Content Moderation with Large Language Models](https://mlsanity.com/paper/2309.14517/) (2023)
- [LifeTox: Unveiling Implicit Toxicity in Life Advice](https://mlsanity.com/paper/2311.09585/) (2023)
- [MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection](https://mlsanity.com/paper/2401.06526/) (2024)
- [Identification of Bias Against People with Disabilities in Sentiment Analysis and Toxicity Detection Models](https://mlsanity.com/paper/2111.13259/) (2021)
- [Legilimens: Practical and Unified Content Moderation for Large Language Model Services](https://mlsanity.com/paper/2408.15488/) (2024)

## Abstract (arXiv)

Moderation is crucial to promoting healthy on-line discussions. Although several `toxicity' detection datasets and models have been published, most of them ignore the context of the posts, implicitly assuming that comments maybe judged independently. We investigate this assumption by focusing on two questions: (a) does context affect the human judgement, and (b) does conditioning on context improve performance of toxicity detection systems? We experiment with Wikipedia conversations, limiting the notion of context to the previous post in the thread and the discussion title. We find that context can both amplify or mitigate the perceived toxicity of posts. Moreover, a small but significant subset of manually labeled posts (5% in one of our experiments) end up having the opposite toxicity labels if the annotators are not provided with context. Surprisingly, we also find no evidence that context actually improves the performance of toxicity classifiers, having tried a range of classifiers and mechanisms to make them context aware. This points to the need for larger datasets of comments annotated in context. We make our code and data publicly available.