mlsanityData to 14 Oct 2026

The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models

arXiv 2404.16019 ↗First posted 24 Apr 2024cs.CLcited by 144

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human feedback is central to the alignment of Large Language Models (LLMs). However, open questions remain about methods (how), domains (where), people (who) and objectives (to what end) of feedback processes. To navigate these questions, we introduce PRISM, a dataset that maps the sociodemographics and stated preferences of 1,500 diverse participants from 75 countries, to their contextual preferences and fine-grained feedback in 8,011 live conversations with 21 LLMs. With PRISM, we contribute (i) wider geographic and demographic participation in feedback; (ii) census-representative samples for two countries (UK, US); and (iii) individualised ratings that link to detailed participant profiles, permitting personalisation and attribution of sample artefacts. We target subjective and multicultural perspectives on value-laden and controversial issues, where we expect interpersonal and cross-cultural disagreement. We use PRISM in three case studies to demonstrate the need for careful consideration of which humans provide what alignment data.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 163 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

62 of 163 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback16 Apr
  2. 2024Investigating Cultural Alignment of Large Language Models20 Feb
  3. 2024A Roadmap to Pluralistic Alignment7 Feb
  4. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  5. 2023The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization2 Nov
  6. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  7. 2023Human Feedback is not Gold Standard28 Sep
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  9. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  10. 2022Training Language Models with Language Feedback29 Apr

Other references · 52

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Mistral 7Bcited by 3,164
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2024Mixtral of Expertscited by 1,823
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2018Model Cards for Model Reportingcited by 1,080
38 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2023Whose Opinions Do Language Models Reflect?cited by 773
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  10. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  12. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  13. 2022Teaching language models to support answers with verified quotescited by 292
  14. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  15. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  16. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  17. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  18. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  19. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  20. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  21. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  22. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
  23. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  24. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  25. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 61
  26. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  27. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 53
  28. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 53
  29. 2023Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Modelscited by 43
  30. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  31. 2021The Challenge of Value Alignment: from Fairer Algorithms to AI Safetycited by 35
  32. 2024The illusion of artificial inclusioncited by 34
  33. 2024Aligning Crowd Feedback via Distributional Preference Reward Modelingcited by 28
  34. 2024AnthroScore: A Computational Linguistic Measure of Anthropomorphismcited by 27
  35. 2016Continuously Learning Neural Dialogue Managementcited by 12
  36. 2023Evaluating Agents using Social Choice Theorycited by 11
  37. 2018Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographicscited by 10
  38. 2024Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefitscited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 50; 2026: 73.

144 papers.

Sort:

By area:

  1. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  2. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  3. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  4. 2024Can LLM be a Personalized Judge?cited by 100
  5. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  6. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 61
  7. 2025Reinforcement Learning from Human Feedbackcited by 58
  8. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 58
  9. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 52
  10. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  11. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  12. 2026HumanLM: Simulating Users with State Alignment Beats Response Imitationcited by 37
38 more
  1. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  2. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  3. 2025IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistancecited by 26
  4. 2025On Benchmarking Human-Like Intelligence in Machinescited by 25
  5. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  6. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 22
  7. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 21
  8. 2025AI Alignment at Your Discretioncited by 20
  9. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  10. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
  11. 2025PRIME: Large Language Model Personalization with Cognitive Dual-Memory and Personalized Thought Processcited by 18
  12. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 16
  13. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 15
  14. 2025The AI Gap: How Socioeconomic Status Affects Language Technology Interactionscited by 15
  15. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  16. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14
  17. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  18. 2025Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lenscited by 14
  19. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 14
  20. 2026OdysSim: Building Foundation Models for Human Behavior Simulationcited by 13
  21. 2026Where does output diversity collapse in post-training?cited by 11
  22. 2024A SMART Mnemonic Sounds like "Glue Tonic": Mixing LLMs with Student Feedback to Make Mnemonic Learning Stickcited by 10
  23. 2024The Future of Open Human Feedbackcited by 10
  24. 2025From tools to thieves: Measuring and understanding public perceptions of AI through crowdsourced metaphorscited by 10
  25. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  26. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 10
  27. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 10
  28. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 10
  29. 2025Statistical Multicriteria Evaluation of LLM-Generated Textcited by 9
  30. 2024The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Communitycited by 8
  31. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  32. 2025ProRefine: Inference-Time Prompt Refinement with Textual Feedbackcited by 8
  33. 2025Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personascited by 8
  34. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 7
  35. 2025Mapping Geopolitical Bias in 11 Large Language Models: A Bilingual, Dual-Framing Analysis of U.S.-China Tensionscited by 7
  36. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 7
  37. 2025Benchmarking Overton Pluralism in LLMscited by 7
  38. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs
  2. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs
  3. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset
  4. 2025On the Alignment of Large Language Models with Global Human Opinion
  5. 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
  6. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
  7. 2026Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Models
  8. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior