# LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans

arXiv: https://arxiv.org/abs/2604.19787 · HTML: https://mlsanity.com/paper/2604.19787/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 23 of 23)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Simulating Opinion Dynamics with Networks of LLM-based Agents](https://mlsanity.com/paper/2311.09618/) (2023)
- [OASIS: Open Agent Social Interaction Simulations with One Million Agents](https://mlsanity.com/paper/2411.11581/) (2024)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [Evaluating Large Language Model Biases in Persona-Steered Generation](https://mlsanity.com/paper/2405.20253/) (2024)
- [Large language models surpass human experts in predicting neuroscience results](https://mlsanity.com/paper/2403.03230/) (2024)
- [Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms](https://mlsanity.com/paper/2310.05984/) (2023)
- [Large Language Models Can Infer Psychological Dispositions of Social Media Users](https://mlsanity.com/paper/2309.08631/) (2023)
- [Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models](https://mlsanity.com/paper/2401.07115/) (2024)
- [Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality](https://mlsanity.com/paper/2306.04308/) (2023)
- [Does GPT-4 surpass human performance in linguistic pragmatics?](https://mlsanity.com/paper/2312.09545/) (2023)
- [Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension](https://mlsanity.com/paper/2404.08885/) (2024)
- [Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism](https://mlsanity.com/paper/2506.21974/) (2025)
- [Cross-Platform Social Dynamics: An Analysis of ChatGPT and COVID-19 Vaccine Conversations](https://mlsanity.com/paper/2310.11116/) (2023)
- [Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction](https://mlsanity.com/paper/2602.22752/) (2026)
- [Bots as Virtual Confederates: Design and Ethics](https://mlsanity.com/paper/1611.00447/) (2016)

## Built on by (most cited first, 1 of 1)

- [Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench](https://mlsanity.com/paper/2605.17079/) (2026)

## Abstract (arXiv)

Social media platforms mediate how billions form opinions and engage with public discourse. As autonomous AI agents increasingly participate in these spaces, understanding their behavioral fidelity becomes critical for platform governance and democratic resilience. Previous work demonstrates that LLM-powered agents can replicate aggregate survey responses, yet few studies test whether agents can predict specific individuals' reactions to specific content. This study benchmarks LLM-based agents' accuracy in predicting human social media reactions (like, dislike, comment, share, no reaction) across 120,000+ unique agent-persona combinations derived from 1,511 Serbian participants and 27 large language models. In Study 1, agents achieved 70.7% overall accuracy, with LLM choice producing a 13 percentage-point performance spread. Study 2 employed binary forced-choice (like/dislike) evaluation with chance-corrected metrics. Agents achieved Matthews Correlation Coefficient (MCC) of 0.29, indicating genuine predictive signal beyond chance. However, conventional text-based supervised classifiers using TF-IDF representations outperformed LLM agents (MCC of 0.36), suggesting predictive gains reflect semantic access rather than uniquely agentic reasoning. The genuine predictive validity of zero-shot persona-prompted agents warns against potential manipulation through easily deploying swarms of behaviorally distinct AI agents on social media, while simultaneously offering opportunities to use such agents in simulations for predicting polarization dynamics and informing AI policy. The advantage of using zero-shot agents is that they require no task-specific training, making their large-scale deployment easy across diverse contexts. Limitations include single-country sampling. Future research should explore multilingual testing and fine-tuning approaches.