# Re-examining Sexism and Misogyny Classification with Annotator Attitudes arXiv: https://arxiv.org/abs/2410.03543 ยท HTML: https://mlsanity.com/paper/2410.03543/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 103 references that are in the map. - First posted on arXiv 4 Oct 2024. ## Builds on (most cited first, 30 of 35) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017) - [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022) - [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019) - [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021) - [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021) - [OpenPrompt: An Open-source Framework for Prompt-learning](https://mlsanity.com/paper/2111.01998/) (2021) - [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021) - [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023) - [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023) - [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023) - [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://mlsanity.com/paper/2109.13563/) (2021) - [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022) - [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021) - [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021) - [SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)](https://mlsanity.com/paper/2304.14803/) (2023) - [When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset](https://mlsanity.com/paper/2306.06826/) (2023) - [Probing LLMs for hate speech detection: strengths and vulnerabilities](https://mlsanity.com/paper/2310.12860/) (2023) - ["Call me sexist, but...": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples](https://mlsanity.com/paper/2004.12764/) (2020) - [SemEval-2023 Task 10: Explainable Detection of Online Sexism](https://mlsanity.com/paper/2303.04222/) (2023) - [ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI](https://mlsanity.com/paper/2109.09483/) (2021) - [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023) - [Handling and Presenting Harmful Text in NLP Research](https://mlsanity.com/paper/2204.14256/) (2022) - [SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection](https://mlsanity.com/paper/2108.03070/) (2021) - [Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection](https://mlsanity.com/paper/2106.15896/) (2021) - [In-Context Learning for Text Classification with Many Labels](https://mlsanity.com/paper/2309.10954/) (2023) - [Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates](https://mlsanity.com/paper/2312.06861/) (2023) - [People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection](https://mlsanity.com/paper/2311.01270/) (2023) - [Multi-label Categorization of Accounts of Sexism using a Neural Framework](https://mlsanity.com/paper/1910.04602/) (2019) - [Incorporating Worker Perspectives into MTurk Annotation Practices for NLP](https://mlsanity.com/paper/2311.02802/) (2023) ## Built on by (most cited first, 8 of 8) - [Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions](https://mlsanity.com/paper/2502.20897/) (2025) - [Explaining Matters: Leveraging Definitions and Semantic Expansion for Sexism Detection](https://mlsanity.com/paper/2506.06238/) (2025) - [Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025](https://mlsanity.com/paper/2606.02255/) (2026) - [Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions](https://mlsanity.com/paper/2509.08217/) (2025) - [Taking a SEAT: Predicting Value Interpretations from Sentiment, Emotion, Argument, and Topic Annotations](https://mlsanity.com/paper/2510.01976/) (2025) - [BioSentinel at EXIST 2026: Soft-Label Optimization with XLM-RoBERTa for Sexism Intent Classification in Memes](https://mlsanity.com/paper/2607.24137/) (2026) - [Reliability-Aware Sexism Detection: Combining DPO with Annotator Agreement and Token-Level Confidence Scoring](https://mlsanity.com/paper/2608.12330/) (2026) - [GENDIS: Gender-Related Discourse and Stance in Multilingual YouTube Comments from India, Kenya, and Nigeria](https://mlsanity.com/paper/2610.17778/) (2026) ## Abstract (arXiv) Gender-Based Violence (GBV) is an increasing problem online, but existing datasets fail to capture the plurality of possible annotator perspectives or ensure the representation of affected groups. We revisit two important stages in the moderation pipeline for GBV: (1) manual data labelling; and (2) automated classification. For (1), we examine two datasets to investigate the relationship between annotator identities and attitudes and the responses they give to two GBV labelling tasks. To this end, we collect demographic and attitudinal information from crowd-sourced annotators using three validated surveys from Social Psychology. We find that higher Right Wing Authoritarianism scores are associated with a higher propensity to label text as sexist, while for Social Dominance Orientation and Neosexist Attitudes, higher scores are associated with a negative tendency to do so. For (2), we conduct classification experiments using Large Language Models and five prompting strategies, including infusing prompts with annotator information. We find: (i) annotator attitudes affect the ability of classifiers to predict their labels; (ii) including attitudinal information can boost performance when we use well-structured brief annotator descriptions; and (iii) models struggle to reflect the increased complexity and imbalanced classes of the new label sets.