# Evaluating the Realism of LLM-powered Social Agents: A Case Study of Reactions to Spanish Online News

arXiv: https://arxiv.org/abs/2605.28598 · HTML: https://mlsanity.com/paper/2605.28598/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [OASIS: Open Agent Social Interaction Simulations with One Million Agents](https://mlsanity.com/paper/2411.11581/) (2024)
- [The spread of low-credibility content by social bots](https://mlsanity.com/paper/1707.07592/) (2017)
- [TweetNLP: Cutting-Edge Natural Language Processing for Social Media](https://mlsanity.com/paper/2206.14774/) (2022)
- [Y Social: an LLM-powered Social Media Digital Twin](https://mlsanity.com/paper/2408.00818/) (2024)
- [Benchmarking Linguistic Diversity of Large Language Models](https://mlsanity.com/paper/2412.10271/) (2024)
- [MOSAIC: Modeling Social AI for Content Dissemination and Regulation in Multi-Agent Simulations](https://mlsanity.com/paper/2504.07830/) (2025)
- [Bots increase exposure to negative and inflammatory content in online social systems](https://mlsanity.com/paper/1802.07292/) (2018)
- [BotSim: LLM-Powered Malicious Social Botnet Simulation](https://mlsanity.com/paper/2412.13420/) (2024)
- ["I'm in the Bluesky Tonight": Insights from a Year Worth of Social Data](https://mlsanity.com/paper/2404.18984/) (2024)
- [Characterizing LLM-driven Social Network: The Chirper.ai Case](https://mlsanity.com/paper/2504.10286/) (2025)
- [Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity](https://mlsanity.com/paper/2507.00657/) (2025)
- [Spotting political social bots in Twitter: A use case of the 2019 Spanish general election](https://mlsanity.com/paper/2004.00931/) (2020)
- [Synthetic generation of online social networks through homophily](https://mlsanity.com/paper/2509.02762/) (2025)
- [Agent-based simulation of online social networks and disinformation](https://mlsanity.com/paper/2512.22082/) (2025)

## Built on by (most cited first, 1 of 1)

- [Should LLM Agents Decide in Social Simulations? Comparing Finite-State and LLM-Based Decision Policies](https://mlsanity.com/paper/2606.12369/) (2026)

## Abstract (arXiv)

LLM-powered social agents are increasingly used to simulate online social behavior, yet their realism remains difficult to validate. Existing work has largely relied on general-purpose benchmarks, while less attention has been paid to short, reactive discourse such as audience replies to online news. In this paper, we evaluate whether LLM-generated reactions to Spanish online news reproduce measurable properties of real audience discourse. Using the Hatemedia dataset, we pair 5,631 news items with 58,555 real audience reactions, and generate a matched synthetic dataset using five LLMs under a shared experimental setting. We compare real and synthetic reactions across three dimensions: hate speech, sentiment, and semantic alignment, considering both off-the-shelf and fine-tuned generation. Results show that off-the-shelf models are poor proxies for real audience reactions: they strongly underproduce hate speech, introduce model-specific sentiment biases, and remain distributionally distant from human replies. Fine-tuning improves fidelity unevenly. Qwen3 provides the most balanced approximation, while Mistral7B achieves the strongest sentiment and semantic alignment but overshoots hate prevalence. Plausible synthetic replies do not necessarily reproduce the distributional properties of public discourse.