mlsanityData to 14 Oct 2026

Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models

arXiv 2506.08593 ↗First posted 10 Jun 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Hate speech detection is a socially sensitive and inherently subjective task, with judgments often varying based on personal traits. While prior work has examined how socio-demographic factors influence annotation, the impact of personality traits on Large Language Models (LLMs) remains largely unexplored. In this paper, we present the first comprehensive study on the role of persona prompts in hate speech classification, focusing on MBTI-based traits. A human annotation survey confirms that MBTI dimensions significantly affect labeling behavior. Extending this to LLMs, we prompt four open-source models with MBTI personas and evaluate their outputs across three hate speech datasets. Our analysis uncovers substantial persona-driven variation, including inconsistencies with ground truth, inter-persona disagreement, and logit-level biases. These findings highlight the need to carefully define persona prompts in LLM-based annotation workflows, with implications for fairness and alignment with human values.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 65 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

38 of 65 references.

References · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  4. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  5. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  6. 2022Using cognitive psychology to understand GPT-3cited by 398
  7. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  8. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  9. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  10. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  11. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  12. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
26 more
  1. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  2. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  3. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  4. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  5. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  6. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  7. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 130
  8. 2024Can LLM be a Personalized Judge?cited by 100
  9. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
  10. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 79
  11. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  12. 2022Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotationcited by 69
  13. 2024Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactionscited by 51
  14. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  15. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  16. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 46
  17. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 45
  18. 2024CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worldscited by 38
  19. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 29
  20. 2024Hate Personified: Investigating the role of LLMs in content moderationcited by 25
  21. 2024Annotation alignment: Comparing LLM and human annotations of conversational safetycited by 23
  22. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  23. 2023ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variationcited by 17
  24. 2024The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detectioncited by 15
  25. 2022AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detectioncited by 12
  26. 2024Can Large Language Models Understand You Better? An MBTI Personality Detection Dataset Aligned with Population Traitscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Modelscited by 0