mlsanityData to 7 Oct 2026

Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions

arXiv 2502.16761 ↗First posted 24 Feb 2025cs.CLcited by 64

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) present novel opportunities in public opinion research by predicting survey responses in advance during the early stages of survey design. Prior methods steer LLMs via descriptions of subpopulations as LLMs' input prompt, yet such prompt engineering approaches have struggled to faithfully predict the distribution of survey responses from human subjects. In this work, we propose directly fine-tuning LLMs to predict response distributions by leveraging unique structural characteristics of survey data. To enable fine-tuning, we curate SubPOP, a significantly scaled dataset of 3,362 questions and 70K subpopulation-response pairs from well-established public opinion surveys. We show that fine-tuning on SubPOP greatly improves the match between LLM predictions and human responses across various subpopulations, reducing the LLM-human gap by up to 46% compared to baselines, and achieves strong generalization to unseen surveys and subpopulations. Our findings highlight the potential of survey-based fine-tuning to improve opinion prediction for diverse, real-world subpopulations and therefore enable more efficient survey designs. Our code is available at https://github.com/JosephJeesungSuh/subpop.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 47 of its 75 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

47 of 75 references.

References · 47

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Mistral 7Bcited by 3,065
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  10. 2023Whose Opinions Do Language Models Reflect?cited by 759
  11. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
35 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  3. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  4. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  5. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  6. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 235
  7. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  8. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  9. 2024A Roadmap to Pluralistic Alignmentcited by 175
  10. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  11. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  12. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  13. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  14. 2023Aligning Language Models to User Opinionscited by 98
  15. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  16. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 71
  17. 2023Questioning the Survey Responses of Large Language Modelscited by 70
  18. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  19. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
  20. 2023On the steerability of large language models toward data-driven personascited by 53
  21. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  22. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 50
  23. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
  24. 2024Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinementcited by 47
  25. 2023AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Predictioncited by 46
  26. 2023Diminished Diversity-of-Thought in a Standard Large Language Modelcited by 43
  27. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  28. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  29. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
  30. 2023Language Models Trained on Media Diets Can Predict Public Opinioncited by 30
  31. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 30
  32. 2024'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participantscited by 27
  33. 2024Distributional Preference Alignment of LLMs via Optimal Transportcited by 26
  34. 2024No Preference Left Behind: Group Distributional Preference Optimizationcited by 21
  35. 2024COMMUNITY-CROSS-INSTRUCT: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communitiescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 41.

64 papers.

Sort:

By area:

  1. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  2. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  3. 2025The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Modelscited by 40
  4. 2025Finetuning LLMs for Human Behavior Prediction in Social Science Experimentscited by 35
  5. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 31
  6. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 31
  7. 2026HumanLM: Simulating Users with State Alignment Beats Response Imitationcited by 31
  8. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 26
  9. 2025Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectificationcited by 17
  10. 2025The threat of analytic flexibility in using large language models to simulate human datacited by 13
  11. 2025Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Modelscited by 8
  12. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoningcited by 5
38 more
  1. 2025Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributionscited by 5
  2. 2025German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studiescited by 4
  3. 2026Beyond Marginal Distributions: A Framework to Evaluate the Representativeness of Demographic-Aligned LLMscited by 4
  4. 2026Identity, Cooperation and Framing Effects within Groups of Real and Simulated Humanscited by 4
  5. 2026Alignment Makes Language Models Normative, Not Descriptivecited by 4
  6. 2026Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approachcited by 4
  7. 2025Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuningcited by 3
  8. 2025Addressing Longstanding Challenges in Cognitive Science with Language Modelscited by 3
  9. 2025ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluationcited by 2
  10. 2025Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Studycited by 2
  11. 2025AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveyscited by 2
  12. 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?cited by 2
  13. 2026Parametric Social Identity Injection and Diversification in Public Opinion Simulationcited by 2
  14. 2026SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulationcited by 2
  15. 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 2
  16. 2026When Can Digital Personas Reliably Approximate Human Survey Findings?cited by 2
  17. 2026The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisionscited by 2
  18. 2025Large Language Models Can Be a Viable Substitute for Expert Political Surveys When a Shock Disrupts Traditional Measurement Approachescited by 1
  19. 2025Synthia: Scalable Grounded Persona Generation from Social Media Datacited by 1
  20. 2025Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Managementcited by 1
  21. 2025Graph-Based Alternatives to LLMs for Human Simulationcited by 1
  22. 2026Large language models accurately predict public perceptions of support for climate action worldwidecited by 1
  23. 2026LLM-SAA: LLM-persona Generated Distributions for Decision-makingcited by 1
  24. 2026Multi-Perspective LLM Annotations for Valid Analyses in Subjective Taskscited by 1
  25. 2026Interview-Informed Generative Agents for Product Discovery: A Validation Studycited by 1
  26. 2026The Collapse of Heterogeneity in Silicon Philosopherscited by 1
  27. 2026Improving Cross-Cultural Survey Simulation with Calibrated Value Personascited by 1
  28. 2026In-Context Learning for the Imputation of Public Opinion Data with Large Language Modelscited by 1
  29. 2026Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inferencecited by 1
  30. 2026Silicon Sampling via Cross-Survey Transfercited by 1
  31. 2026Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judgingcited by 1
  32. 2025Community-Aligned Behavior Under Uncertainty: Evidence of Epistemic Stance Transfer in LLMscited by 0
  33. 2026RADIUS: Ranking, Distribution, and Significance - A Comprehensive Alignment Suite for Survey Simulationcited by 0
  34. 2026Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issuescited by 0
  35. 2026Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agentscited by 0
  36. 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudescited by 0
  37. 2026MEMENTO: Leveraging Web as a Learning Signal for Low-Data Domainscited by 0
  38. 2026Synthetic Personalities: How Well Can LLMs Mimic Individual Respondents Using Socio-Economic Microdata?cited by 0

50 most cited of 64 All 64 (JSON)