# Human Preferences in Large Language Model Latent Space: A Technical Analysis on the Reliability of Synthetic Data in Voting Outcome Prediction

arXiv: https://arxiv.org/abs/2502.16280 · HTML: https://mlsanity.com/paper/2502.16280/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 22 Feb 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 22 of 22)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements](https://mlsanity.com/paper/2401.06766/) (2024)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)
- [Can Large Language Model Agents Simulate Human Trust Behavior?](https://mlsanity.com/paper/2402.04559/) (2024)
- [POSIX: A Prompt Sensitivity Index For Large Language Models](https://mlsanity.com/paper/2410.02185/) (2024)
- [AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction](https://mlsanity.com/paper/2305.09620/) (2023)
- [The language of prompting: What linguistic properties make a prompt successful?](https://mlsanity.com/paper/2311.01967/) (2023)
- [A Large-Scale Simulation on Large Language Models for Decision-Making in Political Science](https://mlsanity.com/paper/2412.15291/) (2024)
- [United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections](https://mlsanity.com/paper/2409.09045/) (2024)

## Built on by (most cited first, 5 of 5)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models](https://mlsanity.com/paper/2512.07890/) (2025)
- [Reasoning Boosts Opinion Alignment in LLMs](https://mlsanity.com/paper/2603.01214/) (2026)
- [Where are the Humans? A Scoping Review of Fairness in Multi-agent AI Systems](https://mlsanity.com/paper/2604.15078/) (2026)
- [Appearing Legitimate is Not Enough: Interrogating Synthetic Agents in Representational Processes through a Participatory Design Lens](https://mlsanity.com/paper/2608.17099/) (2026)

## Abstract (arXiv)

Generative AI (GenAI) is increasingly used in survey contexts to simulate human preferences. While many research endeavors evaluate the quality of synthetic GenAI data by comparing model-generated responses to gold-standard survey results, fundamental questions about the validity and reliability of using LLMs as substitutes for human respondents remain. Our study provides a technical analysis of how demographic attributes and prompt variations influence latent opinion mappings in large language models (LLMs) and evaluates their suitability for survey-based predictions. Using 14 different models, we find that LLM-generated data fails to replicate the variance observed in real-world human responses, particularly across demographic subgroups. In the political space, persona-to-party mappings exhibit limited differentiation, resulting in synthetic data that lacks the nuanced distribution of opinions found in survey data. Moreover, we show that prompt sensitivity can significantly alter outputs for some models, further undermining the stability and predictiveness of LLM-based simulations. As a key contribution, we adapt a probe-based methodology that reveals how LLMs encode political affiliations in their latent space, exposing the systematic distortions introduced by these models. Our findings highlight critical limitations in AI-generated survey data, urging caution in its use for public opinion research, social science experimentation, and computational behavioral modeling.