# When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels

arXiv: https://arxiv.org/abs/2609.13148 · HTML: https://mlsanity.com/paper/2609.13148/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://mlsanity.com/paper/2502.07068/) (2025)
- [Double/Debiased Machine Learning for Treatment and Causal Parameters](https://mlsanity.com/paper/1608.00060/) (2016)
- [Diagnosing Model Performance Under Distribution Shift](https://mlsanity.com/paper/2303.02011/) (2023)
- [A Review of Generalizability and Transportability](https://mlsanity.com/paper/2102.11904/) (2021)
- [Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals](https://mlsanity.com/paper/1709.04589/) (2017)
- [Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions](https://mlsanity.com/paper/2505.17479/) (2025)
- [Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification](https://mlsanity.com/paper/2510.11408/) (2025)
- [How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective](https://mlsanity.com/paper/2502.17773/) (2025)
- [Digital Twins as Funhouse Mirrors: Five Key Distortions](https://mlsanity.com/paper/2509.19088/) (2025)
- [Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents](https://mlsanity.com/paper/2602.18462/) (2026)
- [Demystifying Prediction Powered Inference](https://mlsanity.com/paper/2601.20819/) (2026)
- [Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas](https://mlsanity.com/paper/2509.22957/) (2025)
- [Fine-Tune, Then Rectify](https://mlsanity.com/paper/2511.19486/) (2025)
- [Beyond Reweighting: On the Predictive Role of Covariate Shift in Effect Generalization](https://mlsanity.com/paper/2412.08869/) (2024)
- [Prediction-Powered Inference with Inverse Probability Weighting](https://mlsanity.com/paper/2508.10149/) (2025)
- [Generalized Prediction-Powered Inference, with Application to Binary Classifier Evaluation](https://mlsanity.com/paper/2602.10332/) (2026)
- [Predicting data value before collection: A coefficient for prioritizing sources under random distribution shift](https://mlsanity.com/paper/2504.06570/) (2025)
- [More Parameters Than Populations: A Systematic Literature Review of Large Language Models within Survey Research](https://mlsanity.com/paper/2509.03391/) (2025)

## Built on by (most cited first, 1 of 1)

- [RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations](https://mlsanity.com/paper/2610.01780/) (2026)

## Abstract (arXiv)

Large language models are increasingly deployed as synthetic consumer panels, promising 97% cost reductions over traditional surveys. Yet aggregate validation metrics conceal systematic failures: variance compression, coefficient sign-flips, subgroup error balloons of 10--30 percentage points, and global corrections that worsen demographic bias. We provide a formal framework for deciding when to trust, correct, or abandon LLM-generated consumer data. The framework decomposes synthetic-panel bias into covariate and concept shift, develops testable diagnostics with interpretable decision thresholds, and supplies a doubly robust AIPW estimator requiring only a small calibration sample (n = 50-300). We validate on three testbeds. In controlled simulations the decision rule achieves 100% accuracy (180/180 replications). On the American National Election Study with pre-existing LLM failures, it correctly flags heterogeneous concept shift and reduces naive bias by 92.9-99.6%. On the Twin-2K-500 consumer pricing dataset (172,884 paired human and GPT-4.1-mini responses), it correctly routes full-sample estimation to Trust and subgroup targeting to Correct, with 83-94% bias reduction.