# Like trainer, like bot? Inheritance of bias in algorithmic content moderation

arXiv: https://arxiv.org/abs/1707.01477 · HTML: https://mlsanity.com/paper/1707.01477/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2017.

## Built on by (most cited first, 30 of 35)

- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis](https://mlsanity.com/paper/2308.16705/) (2023)
- [On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research](https://mlsanity.com/paper/2304.12397/) (2023)
- [Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering](https://mlsanity.com/paper/2211.03433/) (2022)
- [Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Study](https://mlsanity.com/paper/2204.01440/) (2022)
- [Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits](https://mlsanity.com/paper/2102.04256/) (2021)
- [Detecting Community Sensitive Norm Violations in Online Conversations](https://mlsanity.com/paper/2110.04419/) (2021)
- [Safety and Fairness for Content Moderation in Generative Models](https://mlsanity.com/paper/2306.06135/) (2023)
- [Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation](https://mlsanity.com/paper/2310.20470/) (2023)
- [Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science](https://mlsanity.com/paper/2203.07031/) (2022)
- [Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations](https://mlsanity.com/paper/2503.01623/) (2025)
- [AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detection](https://mlsanity.com/paper/2212.10405/) (2022)
- [Dataset of Fake News Detection and Fact Verification: A Survey](https://mlsanity.com/paper/2111.03299/) (2021)
- [Towards Equal Gender Representation in the Annotations of Toxic Language Detection](https://mlsanity.com/paper/2106.02183/) (2021)
- [Towards WinoQueer: Developing a Benchmark for Anti-Queer Bias in Large Language Models](https://mlsanity.com/paper/2206.11484/) (2022)
- [Rater Cohesion and Quality from a Vicarious Perspective](https://mlsanity.com/paper/2408.08411/) (2024)
- [Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions](https://mlsanity.com/paper/2211.11798/) (2022)
- [Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning](https://mlsanity.com/paper/2307.10189/) (2023)
- [Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities](https://mlsanity.com/paper/2311.00203/) (2023)
- [Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback](https://mlsanity.com/paper/2406.05902/) (2024)
- [Mitigation of Diachronic Bias in Fake News Detection Dataset](https://mlsanity.com/paper/2108.12601/) (2021)
- [From Biased Selective Labels to Pseudo-Labels: An Expectation-Maximization Framework for Learning from Biased Decisions](https://mlsanity.com/paper/2406.18865/) (2024)
- [Human-in-the-Loop Hate Speech Classification in a Multilingual Context](https://mlsanity.com/paper/2212.02108/) (2022)
- [Identity-related Speech Suppression in Generative AI Content Moderation](https://mlsanity.com/paper/2409.13725/) (2024)
- [A Framework for Evaluating LLMs Under Task Indeterminacy](https://mlsanity.com/paper/2411.13760/) (2024)
- [Like Article, Like Audience: Enforcing Multimodal Correlations for Disinformation Detection](https://mlsanity.com/paper/2108.13892/) (2021)
- [Addressing contingency in algorithmic (mis)information classification: Toward a responsible machine learning agenda](https://mlsanity.com/paper/2210.09014/) (2022)
- [Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection](https://mlsanity.com/paper/2305.14735/) (2023)
- [Suum Cuique: Studying Bias in Taboo Detection with a Community Perspective](https://mlsanity.com/paper/2203.11401/) (2022)

## Abstract (arXiv)

The internet has become a central medium through which `networked publics' express their opinions and engage in debate. Offensive comments and personal attacks can inhibit participation in these spaces. Automated content moderation aims to overcome this problem using machine learning classifiers trained on large corpora of texts manually annotated for offence. While such systems could help encourage more civil debate, they must navigate inherently normatively contestable boundaries, and are subject to the idiosyncratic norms of the human raters who provide the training data. An important objective for platforms implementing such measures might be to ensure that they are not unduly biased towards or against particular norms of offence. This paper provides some exploratory methods by which the normative biases of algorithmic content moderation systems can be measured, by way of a case study using an existing dataset of comments labelled for offence. We train classifiers on comments labelled by different demographic subsets (men and women) to understand how differences in conceptions of offence between these groups might affect the performance of the resulting models on various test sets. We conclude by discussing some of the ethical choices facing the implementers of algorithmic moderation systems, given various desired levels of diversity of viewpoints amongst discussion participants.