# Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive

arXiv: https://arxiv.org/abs/2301.12534 · HTML: https://mlsanity.com/paper/2301.12534/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 79 references that are in the map.
- First posted on arXiv 29 Jan 2023.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://mlsanity.com/paper/1909.04251/) (2019)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis](https://mlsanity.com/paper/1701.08118/) (2017)
- [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://mlsanity.com/paper/2109.13563/) (2021)
- [Multilingual Offensive Language Identification with Cross-lingual Embeddings](https://mlsanity.com/paper/2010.05324/) (2020)
- [Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings](https://mlsanity.com/paper/1904.01596/) (2019)
- [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021)
- [All You Need is "Love": Evading Hate-speech Detection](https://mlsanity.com/paper/1808.09115/) (2018)
- [Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection](https://mlsanity.com/paper/2103.01664/) (2021)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement](https://mlsanity.com/paper/1808.06080/) (2018)
- [Learning Soft Labels via Meta Learning](https://mlsanity.com/paper/2009.09496/) (2020)
- [Pre-train or Annotate? Domain Adaptation with a Constrained Budget](https://mlsanity.com/paper/2109.04711/) (2021)
- [What is Gab? A Bastion of Free Speech or an Alt-Right Echo Chamber?](https://mlsanity.com/paper/1802.05287/) (2018)
- [We Don't Speak the Same Language: Interpreting Polarization through Machine Translation](https://mlsanity.com/paper/2010.02339/) (2020)
- [Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments](https://mlsanity.com/paper/2203.11764/) (2022)
- [Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence](https://mlsanity.com/paper/2105.12936/) (2021)
- [#Election2020: The First Public Twitter Dataset on the 2020 US Presidential Election](https://mlsanity.com/paper/2010.00600/) (2020)
- ["Short is the Road that Leads from Fear to Hate": Fear Speech in Indian WhatsApp Groups](https://mlsanity.com/paper/2102.03870/) (2021)
- [Are Chess Discussions Racist? An Adversarial Hate Speech Data Set](https://mlsanity.com/paper/2011.10280/) (2020)
- [Fringe News Networks: Dynamics of US News Viewership following the 2020 Presidential Election](https://mlsanity.com/paper/2101.10112/) (2021)
- [Neighborhood-based Pooling for Population-level Label Distribution Learning](https://mlsanity.com/paper/2003.07406/) (2020)
- ["A Virus Has No Religion": Analyzing Islamophobia on Twitter During the COVID-19 Outbreak](https://mlsanity.com/paper/2107.05104/) (2021)

## Built on by (most cited first, 16 of 16)

- [Hate Personified: Investigating the role of LLMs in content moderation](https://mlsanity.com/paper/2410.02657/) (2024)
- [Rater Cohesion and Quality from a Vicarious Perspective](https://mlsanity.com/paper/2408.08411/) (2024)
- [ProRefine: Inference-Time Prompt Refinement with Textual Feedback](https://mlsanity.com/paper/2506.05305/) (2025)
- [Value Preferences Estimation and Disambiguation in Hybrid Participatory Systems](https://mlsanity.com/paper/2402.16751/) (2024)
- [When Neutral Summaries are not that Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries](https://mlsanity.com/paper/2410.09978/) (2024)
- [ARTICLE: Annotator Reliability Through In-Context Learning](https://mlsanity.com/paper/2409.12218/) (2024)
- [Infrastructure Ombudsman: Mining Future Failure Concerns from Structural Disaster Response](https://mlsanity.com/paper/2402.13528/) (2024)
- [LLMs and Finetuning: Benchmarking cross-domain performance for hate speech detection](https://mlsanity.com/paper/2310.18964/) (2023)
- [How Inclusively do LMs Perceive Social and Moral Norms?](https://mlsanity.com/paper/2502.02696/) (2025)
- [Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech](https://mlsanity.com/paper/2502.09004/) (2025)
- [Towards Generalized Offensive Language Identification](https://mlsanity.com/paper/2407.18738/) (2024)
- [PromptAug: Fine-grained Conflict Classification Using Data Augmentation](https://mlsanity.com/paper/2506.22491/) (2025)
- [Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEM](https://mlsanity.com/paper/2604.08425/) (2026)
- [Investigating Vaccine Buyer's Remorse: Post-Vaccination Decision Regret in COVID-19 Social Media Using Politically Diverse Human Annotation](https://mlsanity.com/paper/2604.09626/) (2026)
- [From Self to Other: Evaluating Demographic Perspective-Taking in LLM Hate Speech Annotation](https://mlsanity.com/paper/2606.06266/) (2026)
- [Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment](https://mlsanity.com/paper/2608.30842/) (2026)

## Abstract (arXiv)

Offensive speech detection is a key component of content moderation. However, what is offensive can be highly subjective. This paper investigates how machine and human moderators disagree on what is offensive when it comes to real-world social web political discourse. We show that (1) there is extensive disagreement among the moderators (humans and machines); and (2) human and large-language-model classifiers are unable to predict how other human raters will respond, based on their political leanings. For (1), we conduct a noise audit at an unprecedented scale that combines both machine and human responses. For (2), we introduce a first-of-its-kind dataset of vicarious offense. Our noise audit reveals that moderation outcomes vary wildly across different machine moderators. Our experiments with human moderators suggest that political leanings combined with sensitive issues affect both first-person and vicarious offense. The dataset is available through https://github.com/Homan-Lab/voiced.