mlsanityData to 14 Oct 2026

Personas with Attitudes: Controlling LLMs for Diverse Data Annotation

arXiv 2410.11745 ↗First posted 15 Oct 2024cs.CL cs.HCcited by 21

Not on the map. Why?

Abstract (arXiv)

We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results show that persona-prompted LLMs produce more diverse annotations than LLMs prompted without personas and that these effects are both controllable and repeatable, making our approach a suitable tool for improving data annotation in subjective NLP tasks like toxicity detection.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 17 of its 58 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

17 of 58 references.

References · 17

  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  2. 2023Whose Opinions Do Language Models Reflect?cited by 773
  3. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  4. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  5. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 270
  6. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  7. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  8. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  9. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
  10. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  11. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 52
  12. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 46
5 more
  1. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  2. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  3. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  4. 2023Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variationcited by 6
  5. 2024Artificial Data, Real Insights: Evaluating Opportunities and Risks of Expanding the Data Ecosystem with Synthetic Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 7.

21 papers.

Sort:

By area:

  1. 2025LLM Generated Persona is a Promise with a Catchcited by 81
  2. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 33
  3. 2024Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personascited by 16
  4. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generationcited by 8
  5. 2025SynthTRIPs: A Knowledge-Grounded Framework for Benchmark Query Generation for Personalized Tourism Recommenderscited by 8
  6. 2025Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personascited by 8
  7. 2025The Impact of Persona-based Political Perspectives on Hateful Content Detectioncited by 5
  8. 2025Persona-Based Synthetic Data Generation Using Multi-Stage Conditioning with Large Language Models for Emotion Recognitioncited by 5
  9. 2025Political Ideology Shifts in Large Language Modelscited by 5
  10. 2025German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studiescited by 5
  11. 2026The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Studycited by 5
  12. 2025Algorithmic Fairness in NLP: Persona-Infused LLMs for Human-Centric Hate Speech Detectioncited by 2
9 more
  1. 2026Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfercited by 2
  2. 2026How Well Do Large Language Models Capture Human Personality?cited by 2
  3. 2025Ideology-Based LLMs for Content Moderationcited by 1
  4. 2026Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluationcited by 1
  5. 2025Probing Social Identity Bias in Chinese LLMs with Gendered Pronouns and Social Groupscited by 0
  6. 2025Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settingscited by 0
  7. 2026PartisanLens: A Multilingual Dataset of Hyperpartisan and Conspiratorial Immigration Narratives in European Mediacited by 0
  8. 2026Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Modelscited by 0
  9. 2026LLM-Ideoplasticity: Measuring Ideological Plasticity in the Political Behavior of LLMs as a Context-Conditioned Distributioncited by 0