# Hypothesis Testing for Quantifying LLM-Human Misalignment in Multiple Choice Settings

arXiv: https://arxiv.org/abs/2506.14997 · HTML: https://mlsanity.com/paper/2506.14997/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 6 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Builds on (most cited first, 2 of 2)

- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)

## Built on by (most cited first, 2 of 2)

- [Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation](https://mlsanity.com/paper/2604.03257/) (2026)
- [Stance Drift: How AI-mediated Communication Distorts Our Message](https://mlsanity.com/paper/2610.04620/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) increasingly appear in social science research (e.g., economics and marketing), it becomes crucial to assess how well these models replicate human behavior. In this work, using hypothesis testing, we present a quantitative framework to assess the misalignment between LLM-simulated and actual human behaviors in multiple-choice survey settings. This framework allows us to determine in a principled way whether a specific language model can effectively simulate human opinions, decision-making, and general behaviors represented through multiple-choice options. We applied this framework to a popular language model for simulating people's opinions in various public surveys and found that this model is ill-suited for simulating the tested sub-populations (e.g., across different races, ages, and incomes) for contentious questions. This raises questions about the alignment of this language model with the tested populations, highlighting the need for new practices in using LLMs for social science studies beyond naive simulations of human subjects.