The average-farmer illusion in language-model simulations of agricultural decisions
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
Language-model agents are increasingly used as synthetic people in surveys and social simulations, yet their apparent realism is often judged from population averages or distributional similarity. We tested what such evidence actually establishes by comparing Claude, Codex and Kimi under four prespecified prompt designs with matched farmer decisions from China and four African countries. Some configurations reproduced observed means and adoption rates. However, their person-level predictions were weak; their decisions clustered around typical values and policy-relevant extremes were largely missing. Most strikingly, a simple generator fitted only to the observed marginal dis- tribution, and given no information about any farmer, achieved greater distributional similarity than every language-model configuration. Prompt additions produced conditional gains rather than uni- versal improvement: results varied with model, outcome, population and validation target. We call this the average-farmer illusion: a synthetic population can look realistic while failing to repro- duce who does what or how behaviour varies. We provide a claim-matched validation framework and reusable modular prompts that turn prompt construction into an auditable experimental process. Population-level resemblance should therefore be treated as the start of validation, not as evidence of individual simulation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 26 of its 53 references that are in the map. First posted on arXiv 14 Sep 2026.
Builds on
26 of 53 references.
Closest ancestors · 4
Same direction, newest first
- 2025Leveraging LLM-based agents for social science research: insights from citation network simulations5 Nov
- 2025AI Agent Behavioral Science4 Jun
- 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society12 Feb
- 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals15 Nov
Other references · 22
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2025Kimi K2: Open Agentic Intelligencecited by 567
- 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
- 2024What Large Language Models Know and What People Think They Knowcited by 110
- 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
- 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
- 2024Large Language Models for Disease Diagnosis: A Scoping Reviewcited by 51
10 more
- 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 18
- 2025PMark: Towards Robust and Distortion-free Semantic-level Watermarking with Channel Constraintscited by 16
- 2025Large language models as uncertainty-calibrated optimizers for experimental discoverycited by 13
- 2024Data Generation Using Large Language Models for Text Classification: An Empirical Case Studycited by 11
- 2025Variance reduction in output from generative AIcited by 6
- 2026Is More Context Always Better? Examining LLM Reasoning Capability for Time Interval Predictioncited by 4
- 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
- 2024From Text to Insight: Leveraging Large Language Models for Performance Evaluation in Managementcited by 2
- 2026Context-Value-Action Architecture for Value-Driven Large Language Model Agentscited by 2
- 2026KISS - Knowledge Infrastructure for Scientific Simulation: A Scaffolding for Agentic Earth Sciencecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Benchmarking large language model agent societies against human behavioural distributions
- 2025LLMs Model Non-WEIRD Populations: Experiments with Synthetic Cultural Agents
- 2024Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models
- 2025Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making
- 2025Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations
- 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective
- 2026Multi-dimensional Bias in Modeling Multi-dimensional Preferences: Evaluating the Ability of Synthetic Agents to Replace Human Participants in Conjoint Experiments
- 2026Poor Man's Agentic Modeling: Simulating Large LLM-Agent Societies on a Laptop