mlsanityData to 7 Oct 2026

Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench

arXiv 2605.17079 ↗First posted 16 May 2026cs.CL cs.AI cs.CYcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

LLMs are increasingly used as ``digital consumers'' to simulate public opinion, pre-test marketing decisions, and anticipate audience response. However, existing evaluations rarely ask whether a model can reconstruct the concrete reaction patterns that real consumers surface in public discourse. We introduce ConsumerSimBench, a benchmark built from 1,553 real Chinese social-media topics and 23,122 atomic, rule-audited criteria spanning four reaction families. Rather than scoring open-ended generations with a holistic preference judge, ConsumerSimBench decomposes each task into auditable yes-no decisions over concrete reaction points, raising three-judge agreement from 65.8% to 92.1% with 98.4% agreement between pointwise judge decisions and human-majority labels. Across 13 frontier generators, the strongest model, Gemini-3.1-Pro, covers only 47.8% of real reaction criteria, while GPT-5.2 and Claude-4.6 trail far behind despite their strength on technical benchmarks. The failures reveal a sharp gap between technical-benchmark performance and socially grounded consumer intuition. A direct structured reasoning prompt decreases coverage, while a generate--reflect multi-agent pipeline improves MiMo-V2.5-Pro from 32.9% to 37.6% on a subset. ConsumerSimBench reframes consumer simulation as a forecasting problem over real public-discourse reactions, showing that frontier LLMs remain far from reliably predicting what consumers will actually care about in high-context Chinese consumer discourse.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 36 of its 96 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

36 of 96 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Think Socially via Cognitive Reasoning26 Sep
  2. 2025MotiveBench: How Far Are We From Human-Like Motivational Reasoning in Large Language Models?16 Jun

Other references · 34

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  4. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  5. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  6. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 228
  7. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 180
  8. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  9. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 127
  10. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  11. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  12. 2025SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Userscited by 50
22 more
  1. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  2. 2025TwinMarket: A Scalable Behavioral and Social Simulation for Financial Marketscited by 46
  3. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 36
  4. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  5. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  6. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  7. 2025Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shoppingcited by 16
  8. 2024Social Skill Training with Large Language Modelscited by 13
  9. 2025LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratingscited by 9
  10. 2024Assessing Social Alignment: Do Personality-Prompted Large Language Models Behave Like Humans?cited by 8
  11. 2025Infusing Theory of Mind into Socially Intelligent LLM Agentscited by 7
  12. 2025Benchmarking Overton Pluralism in LLMscited by 7
  13. 2025A Review of Incorporating Psychological Theories in LLMscited by 5
  14. 2025Using Large Language Models to Simulate Human Behavioural Experiments: Port of Marscited by 5
  15. 2025Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understandingcited by 5
  16. 2025DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agentscited by 5
  17. 2025Can AI automatically analyze public opinion? A LLM agents-based agentic pipeline for timely public opinion analysiscited by 2
  18. 2025PsychCounsel-Bench: Evaluating the Psychology Intelligence of Large Language Modelscited by 2
  19. 2026Extracting Consumer Insight from Text: A Large Language Model Approach to Emotion and Evaluation Measurementcited by 2
  20. 2026Sell More, Play Less: Benchmarking LLM Realistic Selling Skillcited by 2
  21. 2025A Cross-Cultural Comparison of LLM-based Public Opinion Simulation: Evaluating Chinese and U.S. Models on Diverse Societiescited by 1
  22. 2026LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humanscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating the Realism of LLM-powered Social Agents: A Case Study of Reactions to Spanish Online News
  2. 2026LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses
  3. 2025Persuasion at Play: Understanding Misinformation Dynamics in Demographic-Aware Human-LLM Interactions
  4. 2025Debiasing International Attitudes: LLM Agents for Simulating US-China Perception Changes
  5. 2026LLMs can persuade only psychologically susceptible humans on societal issues, via trust in AI and emotional appeals, amid logical fallacies
  6. 2025ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding
  7. 2025Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation
  8. 2026Synthetic Data in Marketing Research: How to Evaluate and When to Trust