mlsanityData to 14 Oct 2026

Designing Toxic Content Classification for a Diversity of Perspectives

arXiv 2106.04511 ↗First posted 4 Jun 2021cs.SI cs.CR cs.CYcited by 65

Not on the map. Why?

Abstract (arXiv)

In this work, we demonstrate how existing classifiers for identifying toxic comments online fail to generalize to the diverse concerns of Internet users. We survey 17,280 participants to understand how user expectations for what constitutes toxic content differ across demographics, beliefs, and personal experiences. We find that groups historically at-risk of harassment - such as people who identify as LGBTQ+ or young adults - are more likely to to flag a random comment drawn from Reddit, Twitter, or 4chan as toxic, as are people who have personally experienced harassment in the past. Based on our findings, we show how current one-size-fits-all toxicity classification algorithms, like the Perspective API from Jigsaw, can improve in accuracy by 86% on average through personalized model tuning. Ultimately, we highlight current pitfalls and new design directions that can improve the equity and efficacy of toxic content classifiers for all users.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Jun 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 17; 2024: 20; 2025: 12; 2026: 10.

65 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  3. 2023Auditing large language models: a three-layered approachcited by 137
  4. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  5. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
  6. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  7. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 78
  8. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 60
  9. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 46
  10. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  11. 2022Validating Large Language Models with ReLMcited by 33
  12. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 33
38 more
  1. 2025Value Profiles for Encoding Human Variationcited by 29
  2. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  3. 2024Annotation alignment: Comparing LLM and human annotations of conversational safetycited by 23
  4. 2023Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensivecited by 16
  5. 2023GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectivescited by 16
  6. 2023Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Designcited by 15
  7. 2024LLM Roleplay: Simulating Human-Chatbot Interactioncited by 15
  8. 2023Watch Your Language: Investigating Content Moderation with Large Language Modelscited by 14
  9. 2021Two Contrasting Data Annotation Paradigms for Subjective NLP Taskscited by 12
  10. 2024D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluationcited by 11
  11. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
  12. 2024"Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled Peoplecited by 10
  13. 2025Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findingscited by 10
  14. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  15. 2022Automated Identification of Toxic Code Reviews Using ToxiCRcited by 8
  16. 2024Corpus Considerations for Annotator Modeling and Scalingcited by 8
  17. 2024Rater Cohesion and Quality from a Vicarious Perspectivecited by 8
  18. 2024Re-examining Sexism and Misogyny Classification with Annotator Attitudescited by 8
  19. 2024Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotatorscited by 8
  20. 2023Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairnesscited by 7
  21. 2023Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learningcited by 7
  22. 2025NUTMEG: Separating Signal From Noise in Annotator Disagreementcited by 7
  23. 2024Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedbackcited by 6
  24. 2024ARTICLE: Annotator Reliability Through In-Context Learningcited by 6
  25. 2024SPICA: Retrieving Scenarios for Pluralistic In-Context Alignmentcited by 6
  26. 2023Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!cited by 5
  27. 2024Supporting Human Raters with the Detection of Harmful Content using Large Language Modelscited by 5
  28. 2023A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicitycited by 4
  29. 2024A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressionscited by 4
  30. 2024Classification with Conceptual Safeguardscited by 4
  31. 2024Accurate and Data-Efficient Toxicity Prediction when Annotators Disagreecited by 3
  32. 2024A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreementcited by 3
  33. 2025Protect: Towards Robust Guardrailing Stack for Trustworthy Enterprise LLM Systemscited by 3
  34. 2023Mitigating Voter Attribute Bias for Fair Opinion Aggregationcited by 2
  35. 2024How Many Ratings per Item are Necessary for Reliable Significance Testing?cited by 2
  36. 2025Something Just Like TRuST : Toxicity Recognition of Span and Targetcited by 2
  37. 2024Fairness-Aware Multi-Group Target Detection in Online Discussioncited by 1
  38. 2025Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Taskscited by 1

50 most cited of 65 All 65 (JSON)