# Using Large Language Models to Create AI Personas for Replication, Generalization and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findings

arXiv: https://arxiv.org/abs/2408.16073 · HTML: https://mlsanity.com/paper/2408.16073/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 49 references that are in the map.
- First posted on arXiv 28 Aug 2024.

## Builds on (most cited first, 5 of 5)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [DE-COP: Detecting Copyrighted Content in Language Models Training Data](https://mlsanity.com/paper/2402.09910/) (2024)

## Built on by (most cited first, 4 of 4)

- [Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents](https://mlsanity.com/paper/2502.13012/) (2025)
- [Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Research](https://mlsanity.com/paper/2506.01839/) (2025)
- [Boosting Relational Deep Learning with Pretrained Tabular Models](https://mlsanity.com/paper/2504.04934/) (2025)
- [Sentiment Simulation using Generative AI Agents](https://mlsanity.com/paper/2505.22125/) (2025)

## Abstract (arXiv)

This report analyzes the potential for large language models (LLMs) to expedite accurate replication and generalization of published research about message effects in marketing. LLM-powered participants (personas) were tested by replicating 133 experimental findings from 14 papers containing 45 recent studies published in the Journal of Marketing. For each study, the measures, stimuli, and sampling specifications were used to generate prompts for LLMs to act as unique personas. The AI personas, 19,447 in total across all of the studies, generated complete datasets and statistical analyses were then compared with the original human study results. The LLM replications successfully reproduced 76% of the original main effects (84 out of 111), demonstrating strong potential for AI-assisted replication. The overall replication rate including interaction effects was 68% (90 out of 133). Furthermore, a test of how human results generalized to different participant samples, media stimuli, and measures showed that replication results can change when tests go beyond the parameters of the original human studies. Implications are discussed for the replication and generalizability crises in social science, the acceleration of theory building in media and marketing psychology, and the practical advantages of rapid message testing for consumer products. Limitations of AI replications are addressed with respect to complex interaction effects, biases in AI models, and establishing benchmarks for AI metrics in marketing research.