# Are Large Language Models (LLMs) Good Social Predictors?

arXiv: https://arxiv.org/abs/2402.12620 · HTML: https://mlsanity.com/paper/2402.12620/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2024.

## Built on by (most cited first, 13 of 13)

- [Large Language Models for Market Research: A Data-augmentation Approach](https://mlsanity.com/paper/2412.19363/) (2024)
- [A Simulation System Towards Solving Societal-Scale Manipulation](https://mlsanity.com/paper/2410.13915/) (2024)
- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [Reflections from the 2024 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistry](https://mlsanity.com/paper/2411.15221/) (2024)
- [Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation](https://mlsanity.com/paper/2509.06337/) (2025)
- [HARBOR: Exploring Persona Dynamics in Multi-Agent Competition](https://mlsanity.com/paper/2502.12149/) (2025)
- [Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributions](https://mlsanity.com/paper/2510.21977/) (2025)
- [Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approach](https://mlsanity.com/paper/2604.06663/) (2026)
- [Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management](https://mlsanity.com/paper/2510.03310/) (2025)
- [Parametric Social Identity Injection and Diversification in Public Opinion Simulation](https://mlsanity.com/paper/2603.16142/) (2026)
- [Agentic Time Machine as an Infrastructure for Future-Event Forecasting](https://mlsanity.com/paper/2606.21013/) (2026)
- [The LLM Wears Prada: Analysing Gender Bias and Stereotypes through Online Shopping Data](https://mlsanity.com/paper/2504.01951/) (2025)
- [Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods](https://mlsanity.com/paper/2505.19402/) (2025)

## Abstract (arXiv)

The prediction has served as a crucial scientific method in modern social studies. With the recent advancement of Large Language Models (LLMs), efforts have been made to leverage LLMs to predict the human features in social life, such as presidential voting. These works suggest that LLMs are capable of generating human-like responses. However, we find that the promising performance achieved by previous studies is because of the existence of input shortcut features to the response. In fact, by removing these shortcuts, the performance is reduced dramatically. To further revisit the ability of LLMs, we introduce a novel social prediction task, Soc-PRF Prediction, which utilizes general features as input and simulates real-world social study settings. With the comprehensive investigations on various LLMs, we reveal that LLMs cannot work as expected on social prediction when given general input features without shortcuts. We further investigate possible reasons for this phenomenon that suggest potential ways to enhance LLMs for social prediction.