# From Self to Other: Evaluating Demographic Perspective-Taking in LLM Hate Speech Annotation

arXiv: https://arxiv.org/abs/2606.06266 · HTML: https://mlsanity.com/paper/2606.06266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning](https://mlsanity.com/paper/2009.10277/) (2020)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation](https://mlsanity.com/paper/2509.08825/) (2025)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive](https://mlsanity.com/paper/2301.12534/) (2023)
- [Evaluating how LLM annotations represent diverse views on contentious topics](https://mlsanity.com/paper/2503.23243/) (2025)
- [Multilingual Political Views of Large Language Models: Identification and Steering](https://mlsanity.com/paper/2507.22623/) (2025)

## Abstract (arXiv)

Hate speech detection is inherently subjective: people from different demographic groups perceive the same content very differently. Collecting enough annotations from multiple demographic groups is costly and difficult to scale. Persona-conditioned Large Language Models (models prompted to adopt a specific demographic identity) have been proposed as a way to simulate diverse perspectives at scale. But do they actually reflect how different groups disagree? We evaluate three aspects of human social judgement: (i) whether personas from different groups disagree in human-like ways (inter-group disagreement), (ii) whether they become more sensitive when content targets their own identity (in-group sensitivity), and (iii) whether they can accurately predict how another group would react (vicarious prediction). Our results show that no model consistently captures all three dimensions, and performance is highly model-dependent and does not emerge reliably from minimal identity prompts alone. However, vicarious prompting with Llama 3.1 yields the highest cross-group agreement in most demographic axes and provides the closest overall approximation to human disagreement patterns, indicating that this configuration may provide a more reliable setting for automatic annotation aligned with human judgements.