Data to 7 Oct 2026

A Roadmap to Pluralistic Alignment

arXiv 2402.05070 ↗First posted 7 Feb 2024cs.AI cs.CL cs.IRcited by 175

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

With increased power and prevalence of AI systems, it is ever more critical that AI systems are designed to serve all, i.e., people with diverse values and perspectives. However, aligning models to serve pluralistic human values remains an open research question. In this piece, we propose a roadmap to pluralistic alignment, specifically using language models as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems: 1) Overton pluralistic models that present a spectrum of reasonable responses; 2) Steerably pluralistic models that can steer to reflect certain perspectives; and 3) Distributionally pluralistic models that are well-calibrated to a given population in distribution. We also formalize and discuss three possible classes of pluralistic benchmarks: 1) Multi-objective benchmarks, 2) Trade-off steerable benchmarks, which incentivize models to steer to arbitrary trade-offs, and 3) Jury-pluralistic benchmarks which explicitly model diverse human ratings. We use this framework to argue that current alignment techniques may be fundamentally limited for pluralistic AI; indeed, we highlight empirical evidence, both from our own experiments and from other work, that standard alignment procedures might reduce distributional pluralism in models, motivating the need for further research on pluralistic alignment.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 61 of its 117 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

61 of 117 references.

Closest ancestors · 7

Same direction, newest first

2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications24 Oct
2023Human Feedback is not Gold Standard28 Sep
2023Aligning Language Models to User Opinions24 May
2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
2022Constitutional AI: Harmlessness from AI Feedback15 Dec
2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
2020Aligning AI With Shared Human Values5 Aug

Other references · 54

2020Language Models are Few-Shot Learnerscited by 32,199
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
2022Improving alignment of dialogue agents via targeted human judgementscited by 527
2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
2020Artificial Intelligence, Values and Alignmentcited by 443
2018Scalable agent alignment via reward modeling: a research directioncited by 438
2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434

38 more

2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
2023Role-Play with Large Language Modelscited by 290
2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
2023Can Large Language Models Transform Computational Social Science?cited by 245
2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
2023Large Language Model Guided Tree-of-Thoughtcited by 218
2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145
2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 122
2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
2023Cognitive Reframing of Negative Thoughts through Human-Language Model Interactioncited by 77
2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 71
2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
2023Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithmscited by 64
2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
2023Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuringcited by 53
2023On the steerability of large language models toward data-driven personascited by 53
2023Knowledge of cultural moral norms in large language modelscited by 51
2023Teach LLMs to Personalize -- An Approach inspired by Writing Educationcited by 48
2021Early-stopped neural networks are consistentcited by 34
2020Utility is in the Eye of the User: A Critique of NLP Leaderboardscited by 28
2022The Authenticity Gap in Human Evaluationcited by 24

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 52; 2025: 58; 2026: 65.

175 papers.

Sort:

By area:

2024Programming Refusal with Conditional Activation Steeringcited by 191
2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53

38 more

2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
2024Language Model Alignment in Multilingual Trolley Problemscited by 46
2024Moral Alignment for LLM Agentscited by 39
2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
2025Language Model Personalization via Reward Factorizationcited by 27
2024Evaluating the Prompt Steerability of Large Language Modelscited by 26
2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
2025On Benchmarking Human-Like Intelligence in Machinescited by 24
2024A Survey on Human Preference Learning for Large Language Modelscited by 23
2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
2024Generative Monoculture in Large Language Modelscited by 21
2024No Preference Left Behind: Group Distributional Preference Optimizationcited by 21
2025The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Processcited by 21
2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMscited by 20
2025AI Alignment at Your Discretioncited by 20
2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 20
2024Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipediacited by 18
2024Multilingual Large Language Models: A Systematic Surveycited by 18
2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
2024Plurals: A System for Guiding LLMs Via Simulated Social Ensemblescited by 17
2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 17
2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
2024Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Modelscited by 14
2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learningcited by 14
2025Epistemic Alignment: A Mediating Framework for User-LLM Knowledge Deliverycited by 14
2025Because we have LLMs, we Can and Should Pursue Agentic Interpretabilitycited by 14
2025C3AI: Crafting and Evaluating Constitutions for Constitutional AIcited by 13
2024The Call for Socially Aware Language Technologiescited by 12
2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
2025Simulating Society Requires Simulating Thoughtcited by 12
2025On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignmentcited by 12
2026Autonomous Agents on Blockchains: Standards, Execution Models, and Trust Boundariescited by 12
2024Open-World Evaluation for Retrieving Diverse Perspectivescited by 11

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way