# Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation

arXiv: https://arxiv.org/abs/2609.25010 · HTML: https://mlsanity.com/paper/2609.25010/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Builds on (most cited first, 10 of 10)

- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)
- [When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models](https://mlsanity.com/paper/2311.10054/) (2023)
- [LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments](https://mlsanity.com/paper/2406.02611/) (2024)

## Abstract (arXiv)

Marketers increasingly use large language models (LLMs) as "synthetic personas" to predict how an audience will react to a piece of copy before it ships, encouraged by evidence that profile-conditioned LLMs mimic human samples. But is that prediction actually valid against real behaviour - and does the persona machinery help? We present a sim-to-real validity study using the Upworthy Research Archive - thousands of headline A/B tests on shared real traffic, with measured click-through - as held-out ground truth. We compare a ten-persona panel, grounded in the real audience's demographics, against a no-persona zero-shot baseline that simply asks the model how likely a typical reader is to click. Two findings stand out. First, ground-truth reliability is the binding constraint: most A/B tests have no statistically distinguishable winner, so validity can only be measured on the reliable subset (n = 399). Second, and counter to the persona-simulation premise, persona conditioning degrades predictive validity: the no-persona baseline ranks variants markedly better (Kendall τ = 0.361, a medium effect; top-1 accuracy 49.2%) than the persona panel (τ = 0.084; top-1 34.6%), with non-overlapping confidence intervals. Asking the model directly taps an accurate population-level prior; forcing it to role-play specific personas injects bias and noise. The result replicates across three independent Upworthy splits, holds in direction on a different-domain news dataset, and is robust to seed, prompt phrasing, and model choice - across three Gemini tiers and a different model family (OpenAI gpt-4.1, significant paired gap). The takeaway: for predicting aggregate engagement, a plain LLM ranker beats persona simulation - synthetic personas are not merely a weak predictor, they are worse than not using them. All numbers regenerate from a public, artifact-first replication package.