mlsanityData to 7 Oct 2026

Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

arXiv 2512.00461 ↗First posted 29 Nov 2025cs.CY cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Synthetic personae experiments have become a prominent method in Large Language Model alignment research, yet the representativeness and ecological validity of these personae vary considerably between studies. Through a review of 63 peer-reviewed studies published between 2023 and 2025 in leading NLP and AI venues, we reveal a critical gap: task and population of interest are often underspecified in persona-based experiments, despite personalization being fundamentally dependent on these criteria. Our analysis shows substantial differences in user representation, with most studies focusing on limited sociodemographic attributes and only 35% discussing the representativeness of their LLM personae. Based on our findings, we introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity. Our work provides both a comprehensive assessment of current practices and practical guidelines to improve the rigor and ecological validity of persona-based evaluations in language model alignment research.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 70 references that are in the map. First posted on arXiv 29 Nov 2025.

Builds on

34 of 70 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing13 May
  2. 2023Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoning14 Nov

Other references · 32

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2018Model Cards for Model Reportingcited by 1,068
  3. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  4. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  5. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
  6. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  7. 2024Quantifying the Persona Effect in LLM Simulationscited by 164
  8. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  9. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  10. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  11. 2023On the steerability of large language models toward data-driven personascited by 53
  12. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
20 more
  1. 2023Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systemscited by 37
  2. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 28
  3. 2023LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streamingcited by 27
  4. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  5. 2022Personalized Dialogue Generation with Persona-Adaptive Attentioncited by 24
  6. 2024Holistic Safety and Responsibility Evaluations of Advanced AI Modelscited by 18
  7. 2023Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialoguescited by 17
  8. 2023PRODIGy: a PROfile-based DIalogue Generation datasetcited by 16
  9. 2024Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Datasetcited by 15
  10. 2024Commonsense-augmented Memory Construction and Management in Long-term Conversations via Context-aware Persona Refinementcited by 14
  11. 2024GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancycited by 14
  12. 2024Help Me Reflect: Leveraging Self-Reflection Interface Nudges to Enhance Deliberativeness on Online Deliberation Platformscited by 9
  13. 2024Dialogue Language Model with Large-Scale Persona Data Engineeringcited by 9
  14. 2023P5: Plug-and-Play Persona Prompting for Personalized Response Selectioncited by 8
  15. 2024Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledgecited by 8
  16. 2024Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creatorscited by 5
  17. 2025Social Bias in Popular Question-Answering Benchmarkscited by 5
  18. 2024Beyond Discrete Personas: Personality Modeling Through Journal Intensive Conversationscited by 4
  19. 2023Persona-aware Generative Model for Code-mixed Languagecited by 3
  20. 2023We are what we repeatedly do: Inducing and deploying habitual schemas in persona-based responsescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026The Need for a Socially-Grounded Persona Framework for User Simulationcited by 14
  2. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
  3. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  4. 2026Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworkscited by 3
  5. 2026CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experimentscited by 2
  6. 2026Moral Lenses, Political Coordinates: Towards Ideological Positioning of Morally Conditioned LLMscited by 0
  7. 2026PersonaCite: VoC-Grounded Interviewable Agentic Synthetic AI Personas for Verifiable User and Design Researchcited by 0
  8. 2026Synonymix: Unified Group Personas for Generative Simulationscited by 0
  9. 2026Marginal Alignment Does Not Guarantee Joint-Distribution Fidelity: An Official-Reference Audit of Nemotron-Personas-Korea with Cross-Locale Replicationcited by 0
  10. 2026Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Languagecited by 0
  11. 2026A Diagnostic Framework for AI Agent Behaviorcited by 0
  12. 2026Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panelcited by 0
2 more
  1. 2026Distributional Validity of a Korean Synthetic Persona Panel: Evidence From the Korea Media Panel Surveycited by 0
  2. 2026Cultural Divergence Preservation: Diagnosing Flattening and Caricature in LLM-Simulated Survey Populationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment
  2. 2025Measuring Lexical Diversity of Synthetic Data Generated through Fine-Grained Persona Prompting
  3. 2024LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models
  4. 2026Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation
  5. 2024PersonaGym: Evaluating Persona Agents and LLMs
  6. 2025German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studies
  7. 2026When Persona Attributes Improve Population Alignment in Large Language Models
  8. 2025Population-Aligned Persona Generation for LLM-based Social Simulation