# How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective

arXiv: https://arxiv.org/abs/2502.17773 · HTML: https://mlsanity.com/paper/2502.17773/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2025.

## Built on by (most cited first, 12 of 12)

- [Investigating Human-Aligned Large Language Model Uncertainty](https://mlsanity.com/paper/2503.12528/) (2025)
- [SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation](https://mlsanity.com/paper/2604.07513/) (2026)
- [Allocating Human Oversight in AI-Enabled Analytics](https://mlsanity.com/paper/2604.12497/) (2026)
- [Valid Inference with Synthetic Data via Task Exchangeability](https://mlsanity.com/paper/2606.13629/) (2026)
- [When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels](https://mlsanity.com/paper/2609.13148/) (2026)
- [Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models](https://mlsanity.com/paper/2508.08204/) (2025)
- [Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys](https://mlsanity.com/paper/2604.17267/) (2026)
- [Adaptive Querying with AI Persona Priors](https://mlsanity.com/paper/2605.00696/) (2026)
- [LLM-Powered Virtual Population for Demand Simulation and Pricing](https://mlsanity.com/paper/2606.16183/) (2026)
- [Generating Public Health Responses using Survey-Augmented Large Language Models](https://mlsanity.com/paper/2606.21820/) (2026)
- [Learning a Size-Weight Frontier for Synthetic-Augmented Inference](https://mlsanity.com/paper/2608.28576/) (2026)
- [Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study](https://mlsanity.com/paper/2609.07395/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to simulate survey responses, but synthetic data can be misaligned with the human population, leading to unreliable inference. We develop a general framework that converts LLM-simulated responses into reliable confidence sets for population parameters of human responses, quantifying the uncertainty induced by the human-LLM misalignment. The key design choice is the number of simulated responses: too many produce overly narrow sets with poor coverage, while too few yield overly wide and uninformative sets dominated by stochastic noise. We propose a data-driven approach that adaptively selects the simulation sample size to achieve nominal average-case coverage, regardless of the LLM's simulation fidelity or the confidence set construction procedure. The selected sample size is further shown to reflect the effective human population size that the LLM can represent, providing a quantitative measure of its simulation fidelity. Experiments on real survey datasets reveal heterogeneous simulation fidelity across different LLMs and domains.