# Analyzing Norm Violations in Live-Stream Chat

arXiv: https://arxiv.org/abs/2305.10731 · HTML: https://mlsanity.com/paper/2305.10731/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](https://mlsanity.com/paper/2202.11176/) (2022)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [Toxicity Detection: Does Context Really Matter?](https://mlsanity.com/paper/2006.00998/) (2020)
- [Contextualizing Hate Speech Classifiers with Post-hoc Explanation](https://mlsanity.com/paper/2005.02439/) (2020)
- [Community Interaction and Conflict on the Web](https://mlsanity.com/paper/1803.03697/) (2018)
- [Analyzing the Targets of Hate in Online Social Media](https://mlsanity.com/paper/1603.07709/) (2016)
- [A Just and Comprehensive Strategy for Using NLP to Address Online Abuse](https://mlsanity.com/paper/1906.01738/) (2019)
- [Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study](https://mlsanity.com/paper/1906.01603/) (2019)
- [Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detection](https://mlsanity.com/paper/2103.14916/) (2021)
- [Detecting Community Sensitive Norm Violations in Online Conversations](https://mlsanity.com/paper/2110.04419/) (2021)
- [XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models](https://mlsanity.com/paper/2210.16978/) (2022)
- [Surfacing contextual hate speech words within social media](https://mlsanity.com/paper/1711.10093/) (2017)
- [Extracting Inter-community Conflicts in Reddit](https://mlsanity.com/paper/1808.04405/) (2018)
- [Norm violation in online communities -- A study of Stack Overflow comments](https://mlsanity.com/paper/2004.05589/) (2020)
- [Abusive Language Detection in Heterogeneous Contexts: Dataset Collection and the Role of Supervised Attention](https://mlsanity.com/paper/2105.11119/) (2021)

## Built on by (most cited first, 3 of 3)

- [Legilimens: Practical and Unified Content Moderation for Large Language Model Services](https://mlsanity.com/paper/2408.15488/) (2024)
- [ToxiTwitch: Toward Emote-Aware Hybrid Moderation for Live Streaming Platforms](https://mlsanity.com/paper/2601.15605/) (2026)
- [Harmful Suicide Content Detection](https://mlsanity.com/paper/2407.13942/) (2024)

## Abstract (arXiv)

Toxic language, such as hate speech, can deter users from participating in online communities and enjoying popular platforms. Previous approaches to detecting toxic language and norm violations have been primarily concerned with conversations from online forums and social media, such as Reddit and Twitter. These approaches are less effective when applied to conversations on live-streaming platforms, such as Twitch and YouTube Live, as each comment is only visible for a limited time and lacks a thread structure that establishes its relationship with other comments. In this work, we share the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. We articulate several facets of live-stream data that differ from other forums, and demonstrate that existing models perform poorly in this setting. By conducting a user study, we identify the informational context humans use in live-stream moderation, and train models leveraging context to identify norm violations. Our results show that appropriate contextual information can boost moderation performance by 35%.