mlsanityData to 14 Oct 2026

Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism

arXiv 2506.21974 ↗First posted 27 Jun 2025cs.CLcited by 6

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

The ability of Large Language Models (LLMs) to mimic human behavior triggered a plethora of computational social science research, assuming that empirical studies of humans can be conducted with AI agents instead. Since there have been conflicting research findings on whether and when this hypothesis holds, there is a need to better understand the differences in their experimental designs. We focus on replicating the behavior of social network users with the use of LLMs for the analysis of communication on social networks. First, we provide a formal framework for the simulation of social networks, before focusing on the sub-task of imitating user communication. We empirically test different approaches to imitate user behavior on X in English and German. Our findings suggest that social simulations should be validated by their empirical realism measured in the setting in which the simulation components were fitted. With this paper, we argue for more rigor when applying generative-agent-based modeling for social simulation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 20 of its 32 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

20 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations4 Apr
  2. 2024Political Bias in LLMs: Unaligned Moral Values in Agent-centric Simulations21 Aug
  3. 2023Simulating Opinion Dynamics with Networks of LLM-based Agents16 Nov
  4. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2017Decoupled Weight Decay Regularizationcited by 1,831
  6. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  7. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  8. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  9. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  10. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  11. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  12. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 45
4 more
  1. 2024LLMs generate structurally realistic social networks but overestimate political homophilycited by 33
  2. 2024Network Formation and Dynamics Among Multi-LLMscited by 29
  3. 2024Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Studycited by 14
  4. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Predictioncited by 3
  2. 2026LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humanscited by 1
  3. 2026Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Contentcited by 0
  4. 2026Belief Cascades Drive Persuasion in LLM Agent Networkscited by 0
  5. 2026From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencerscited by 0
  6. 2026Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Contentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Agent-Based Modelling Meets Generative AI in Social Network Simulations
  2. 2025Towards Simulating Social Influence Dynamics with LLM-based Multi-agents
  3. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agents
  4. 2024Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models
  5. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
  6. 2025Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation
  7. 2026MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions
  8. 2026AI Agents Alone Are Not (Yet) Sufficient for Social Simulation