# Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

arXiv: https://arxiv.org/abs/2406.14462 · HTML: https://mlsanity.com/paper/2406.14462/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 75 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Training Millions of Personalized Dialogue Agents](https://mlsanity.com/paper/1809.01984/) (2018)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [LLMs Accelerate Annotation for Medical Information Extraction](https://mlsanity.com/paper/2312.02296/) (2023)
- [Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity](https://mlsanity.com/paper/2209.12106/) (2022)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)
- [Mirages: On Anthropomorphism in Dialogue Systems](https://mlsanity.com/paper/2305.09800/) (2023)
- [Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation](https://mlsanity.com/paper/2306.01183/) (2023)
- [A Personalized Dialogue Generator with Implicit User Persona Detection](https://mlsanity.com/paper/2204.07372/) (2022)
- [Bias in Emotion Recognition with ChatGPT](https://mlsanity.com/paper/2310.11753/) (2023)
- [SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks](https://mlsanity.com/paper/2402.01980/) (2024)
- [Lived Experience Matters: Automatic Detection of Stigma on Social Media Toward People Who Use Substances](https://mlsanity.com/paper/2302.02064/) (2023)

## Built on by (most cited first, 23 of 23)

- [NoveltyBench: Evaluating Language Models for Humanlike Diversity](https://mlsanity.com/paper/2504.05228/) (2025)
- [The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models](https://mlsanity.com/paper/2507.16076/) (2025)
- [Normative Evaluation of Large Language Models with Everyday Moral Dilemmas](https://mlsanity.com/paper/2501.18081/) (2025)
- [Persona-Assigned Large Language Models Exhibit Human-Like Motivated Reasoning](https://mlsanity.com/paper/2506.20020/) (2025)
- [One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization](https://mlsanity.com/paper/2601.18572/) (2026)
- [Using Large Language Models to Categorize Strategic Situations and Decipher Motivations Behind Human Behaviors](https://mlsanity.com/paper/2503.15752/) (2025)
- [Persona-Based Synthetic Data Generation Using Multi-Stage Conditioning with Large Language Models for Emotion Recognition](https://mlsanity.com/paper/2507.13380/) (2025)
- [Language of Thought Shapes Output Diversity in Large Language Models](https://mlsanity.com/paper/2601.11227/) (2026)
- [Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias](https://mlsanity.com/paper/2601.18486/) (2026)
- [Modeling Subjectivity in Cognitive Appraisal with Language Models](https://mlsanity.com/paper/2503.11381/) (2025)
- [Investigating Co-Constructive Behavior of Large Language Models in Explanation Dialogues](https://mlsanity.com/paper/2504.18483/) (2025)
- [Persona Alchemy: Designing, Evaluating, and Implementing Psychologically-Grounded LLM Agents for Diverse Stakeholder Representation](https://mlsanity.com/paper/2505.18351/) (2025)
- [Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?](https://mlsanity.com/paper/2511.21218/) (2025)
- [When Persona Attributes Improve Population Alignment in Large Language Models](https://mlsanity.com/paper/2609.02526/) (2026)
- [Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models](https://mlsanity.com/paper/2506.08593/) (2025)
- [Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibility](https://mlsanity.com/paper/2603.03585/) (2026)
- [Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLM](https://mlsanity.com/paper/2609.04738/) (2026)
- [R.U.Psycho? Robust Unified Psychometric Testing of Language Models](https://mlsanity.com/paper/2503.10229/) (2025)
- [Images Amplify Misinformation Sharing in Vision-Language Models](https://mlsanity.com/paper/2505.13302/) (2025)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [Toward Human-Centered Multi-Agent Systems: Integrating Cognition, Culture, Values, and Cooperation in AI Agents](https://mlsanity.com/paper/2606.08274/) (2026)
- [Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races](https://mlsanity.com/paper/2608.01193/) (2026)
- [Personas Differ from Native-Language Generation: Language Pathways Shape LLM Interpersonal Advice](https://mlsanity.com/paper/2608.30873/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. However, these tasks are highly subjective and dependent on human factors, such as one's environment, attitudes, beliefs, and lived experiences. Thus, it may be the case that employing LLMs (which do not have such human factors) in these tasks results in a lack of variation in data, failing to reflect the diversity of human experiences. In this paper, we examine the role of prompting LLMs with human-like personas and asking the models to answer as if they were a specific human. This is done explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. The LLM personas are then evaluated via (1) subjective annotation task (e.g., detecting toxicity) and (2) a belief generation task, where both tasks are known to vary across human factors. We examine the impact of explicit vs. implicit personas and investigate which human factors LLMs recognize and respond to. Results show that explicit LLM personas show mixed results when reproducing known human biases, but generally fail to demonstrate implicit biases. We conclude that LLMs may capture the statistical patterns of how people speak, but are generally unable to model the complex interactions and subtleties of human perceptions, potentially limiting their effectiveness in social science applications.