mlsanityData to 14 Oct 2026

Are Large Language Models Chameleons? An Attempt to Simulate Social Surveys

arXiv 2405.19323 ↗First posted 29 May 2024cs.CL cs.AI cs.CYcited by 8

Not on the map. Why?

Abstract (arXiv)

Can large language models (LLMs) simulate social surveys? To answer this question, we conducted millions of simulations in which LLMs were asked to answer subjective questions. A comparison of different LLM responses with the European Social Survey (ESS) data suggests that the effect of prompts on bias and variability is fundamental, highlighting major cultural, age, and gender biases. We further discussed statistical methods for measuring the difference between LLM answers and survey data and proposed a novel measure inspired by Jaccard similarity, as LLM-generated responses are likely to have a smaller variance. Our experiments also reveal that it is important to analyze the robustness and variability of prompts before using LLMs to simulate social surveys, as their imitation abilities are approximate at best.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

24 of 46 references.

References · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  6. 2023Whose Opinions Do Language Models Reflect?cited by 773
  7. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  8. 2020Aligning AI With Shared Human Valuescited by 723
  9. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  10. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  11. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  12. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
12 more
  1. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  2. 2023Large Language Model Alignment: A Surveycited by 243
  3. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  4. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  5. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  6. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  7. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  8. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  9. 2024Disclosure and Mitigation of Gender Bias in LLMscited by 49
  10. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 46
  11. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  12. 2023Demonstrations of the Potential of AI-based Political Issue Pollingcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 61
  2. 2024The Impact of Large Language Models in Academia: from Writing to Speakingcited by 18
  3. 2025Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectificationcited by 18
  4. 2025Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responsescited by 13
  5. 2025Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributionscited by 5
  6. 2026BenchBench: Benchmarking Automated Benchmark Generationcited by 5
  7. 2025Prompting for Policy: Forecasting Macroeconomic Scenarios with Synthetic LLM Personascited by 4
  8. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?cited by 1