Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Large language models are increasingly used as proxies for human subjects in social science research, yet external validity requires that synthetic agents faithfully reflect the preferences of target human populations. We introduce *preference reconstruction theory*, a framework that formalizes preference alignment as a representation learning problem: constructing a functional basis of proxy agents and recovering population preferences through weighted aggregation. We implement this via *Prompts to Proxies* (textttP2P), a modular two-stage system. Stage 1 uses structured prompting with entropy-based adaptive sampling to construct a diverse agent pool spanning the latent preference space. Stage 2 employs L1-regularized regression to select a compact ensemble whose aggregate response distributions align with observed data from the target population. textttP2P requires no finetuning and no access to sensitive demographic data, incurring only API inference costs. We validate the approach on 14 waves of the American Trends Panel, achieving an average test MSE of 0.014 across diverse topics at approximately 0.8 USD per survey. We additionally test it on the World Values Survey, demonstrating its potential to generalize across locales. When stress-tested against an SFT-aligned baseline, textttP2P achieves competitive performance using less than 3% of the training data.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 69 references that are in the map. First posted on arXiv 14 Sep 2025.
Builds on
38 of 69 references.
Closest ancestors · 3
Same direction, newest first
- 2024SPICA: Retrieving Scenarios for Pluralistic In-Context Alignment16 Nov
- 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignment25 Jun
- 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models17 Oct
Other references · 35
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
- 2018Scalable agent alignment via reward modeling: a research directioncited by 447
- 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
- 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
- 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 365
- 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
23 more
- 2023AI Alignment: A Comprehensive Surveycited by 263
- 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
- 2023Representation Learning with Large Language Models for Recommendationcited by 155
- 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
- 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
- 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 90
- 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
- 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 74
- 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
- 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
- 2023CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agentscited by 65
- 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 54
- 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 53
- 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 43
- 2024Efficiently Learning at Test-Time: Active Fine-Tuning of LLMscited by 42
- 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 16
- 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learningcited by 14
- 2025Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regressioncited by 7
- 2024Policy Prototyping for LLMs: Pluralistic Alignment via Interactive and Collaborative Policymakingcited by 6
- 2025Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equitycited by 4
- 2024Composite Active Learning: Towards Multi-Domain Active Learning with Theoretical Guaranteescited by 3
- 2025Personalizing Large Language Models using Retrieval Augmented Generation and Knowledge Graphcited by 3
- 2025Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversationscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Spectral Souping: A Unified Framework for Online Preference Alignment
- 2026Response Time Enhances Alignment with Heterogeneous Preferences
- 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
- 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
- 2025Robust Preference Alignment via Directional Neighborhood Consensus
- 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
- 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
- 2025Learning a Canonical Basis of Human Preferences from Binary Ratings