# Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench

arXiv: https://arxiv.org/abs/2605.17079 · HTML: https://mlsanity.com/paper/2605.17079/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 36 of its 96 references that are in the map.
- First posted on arXiv 16 May 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 36)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [S3: Social-network Simulation System with Large Language Model-Empowered Agents](https://mlsanity.com/paper/2307.14984/) (2023)
- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars](https://mlsanity.com/paper/2311.17227/) (2023)
- [OASIS: Open Agent Social Interaction Simulations with One Million Agents](https://mlsanity.com/paper/2411.11581/) (2024)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users](https://mlsanity.com/paper/2504.10157/) (2025)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [TwinMarket: A Scalable Behavioral and Social Simulation for Financial Markets](https://mlsanity.com/paper/2502.01506/) (2025)
- [Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations](https://mlsanity.com/paper/2601.17087/) (2026)
- [LLM-based Human Simulations Have Not Yet Been Reliable](https://mlsanity.com/paper/2501.08579/) (2025)
- [Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective](https://mlsanity.com/paper/2408.04638/) (2024)
- [Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping](https://mlsanity.com/paper/2510.07230/) (2025)
- [HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models](https://mlsanity.com/paper/2506.03922/) (2025)
- [Social Skill Training with Large Language Models](https://mlsanity.com/paper/2404.04204/) (2024)
- [LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings](https://mlsanity.com/paper/2510.08338/) (2025)
- [Assessing Social Alignment: Do Personality-Prompted Large Language Models Behave Like Humans?](https://mlsanity.com/paper/2412.16772/) (2024)
- [Infusing Theory of Mind into Socially Intelligent LLM Agents](https://mlsanity.com/paper/2509.22887/) (2025)
- [Benchmarking Overton Pluralism in LLMs](https://mlsanity.com/paper/2512.01351/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understanding](https://mlsanity.com/paper/2510.04899/) (2025)
- [DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents](https://mlsanity.com/paper/2510.25110/) (2025)
- [Using Large Language Models to Simulate Human Behavioural Experiments: Port of Mars](https://mlsanity.com/paper/2506.05555/) (2025)
- [MotiveBench: How Far Are We From Human-Like Motivational Reasoning in Large Language Models?](https://mlsanity.com/paper/2506.13065/) (2025)
- [Sell More, Play Less: Benchmarking LLM Realistic Selling Skill](https://mlsanity.com/paper/2604.07054/) (2026)

## Built on by (most cited first, 1 of 1)

- [Predicting consumer-technology ownership without a diffusion history](https://mlsanity.com/paper/2608.12344/) (2026)

## Abstract (arXiv)

LLMs are increasingly used as ``digital consumers'' to simulate public opinion, pre-test marketing decisions, and anticipate audience response. However, existing evaluations rarely ask whether a model can reconstruct the concrete reaction patterns that real consumers surface in public discourse. We introduce ConsumerSimBench, a benchmark built from 1,553 real Chinese social-media topics and 23,122 atomic, rule-audited criteria spanning four reaction families. Rather than scoring open-ended generations with a holistic preference judge, ConsumerSimBench decomposes each task into auditable yes-no decisions over concrete reaction points, raising three-judge agreement from 65.8% to 92.1% with 98.4% agreement between pointwise judge decisions and human-majority labels. Across 13 frontier generators, the strongest model, Gemini-3.1-Pro, covers only 47.8% of real reaction criteria, while GPT-5.2 and Claude-4.6 trail far behind despite their strength on technical benchmarks. The failures reveal a sharp gap between technical-benchmark performance and socially grounded consumer intuition. A direct structured reasoning prompt decreases coverage, while a generate--reflect multi-agent pipeline improves MiMo-V2.5-Pro from 32.9% to 37.6% on a subset. ConsumerSimBench reframes consumer simulation as a forecasting problem over real public-discourse reactions, showing that frontier LLMs remain far from reliably predicting what consumers will actually care about in high-context Chinese consumer discourse.