mlsanityData to 7 Oct 2026

One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity

arXiv 2411.04427 ↗First posted 7 Nov 2024cs.CLcited by 57

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Researchers in social science and psychology have recently proposed using large language models (LLMs) as replacements for humans in behavioral research. In addition to arguments about whether LLMs accurately capture population-level patterns, this has raised questions about whether LLMs capture human-like conceptual diversity. Separately, it is debated whether post-training alignment (RLHF or RLAIF) affects models' internal diversity. Inspired by human studies, we use a new way of measuring the conceptual diversity of synthetically-generated LLM "populations" by relating the internal variability of simulated individuals to the population-level variability. We use this approach to evaluate non-aligned and aligned LLMs on two domains with rich human behavioral data. While no model reaches human-like diversity, aligned models generally display less diversity than their instruction fine-tuned counterparts. Our findings highlight potential trade-offs between increasing models' value alignment and decreasing the diversity of their conceptual representations.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 43 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

24 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Beyond Preferences in AI Alignment30 Aug

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  9. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  10. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  11. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  12. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
11 more
  1. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  2. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  3. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  4. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 114
  5. 2023Turning large language models into cognitive modelscited by 66
  6. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  7. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  8. 2023Large language models predict human sensory judgments across six modalitiescited by 47
  9. 2024Psychometric Alignment: Capturing Human Knowledge Distributions via Language Modelscited by 18
  10. 2024Bayesian Statistical Modeling with Predictors from LLMscited by 6
  11. 2021Shades of confusion: Lexical uncertainty modulates ad hoc coordination in an interactive communication taskcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 30.

57 papers.

Sort:

By area:

  1. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  2. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 66
  3. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  4. 2025Diverse Preference Optimizationcited by 48
  5. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  6. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  7. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 35
  8. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  9. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  10. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 26
  11. 2025Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agentscited by 22
  12. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
38 more
  1. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
  2. 2025What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Modelscited by 14
  3. 2025The Geometry of Self-Verification in a Task-Specific Reasoning Modelcited by 12
  4. 2025Levels of Analysis for Large Language Modelscited by 9
  5. 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
  6. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 7
  7. 2025What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversitycited by 7
  8. 2025Whose Truth? Pluralistic Geo-Alignment for (Agentic) AIcited by 6
  9. 2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 5
  10. 2026Annotations Mitigate Post-Training Mode Collapsecited by 4
  11. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  12. 2025Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibilitycited by 2
  13. 2025Signatures of human-like processing in Transformer forward passescited by 2
  14. 2026Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMscited by 2
  15. 2026AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writingcited by 2
  16. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  17. 2026PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agentscited by 2
  18. 2025IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Researchcited by 1
  19. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safetycited by 1
  20. 2025Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Modelscited by 1
  21. 2026The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safetycited by 1
  22. 2026Quality-constrained Entropy Maximization Policy Optimization for LLM Diversitycited by 1
  23. 2026Diversity Without Fidelity: A Solver-Sampler Mismatch in Multi-Agent LLM Negotiation Simulationcited by 1
  24. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  25. 2026Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fictioncited by 1
  26. 2026Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMscited by 1
  27. 2026A Multi-Timescale Recursive Self-Improvement Engine for Open-Ended Persona Growthcited by 1
  28. 2026Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Modelscited by 1
  29. 2025Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policiescited by 0
  30. 2026Detecting RLVR Training Data via Structural Convergence of Reasoningcited by 0
  31. 2026Geography According to ChatGPT -- How Generative AI Represents and Reasons about Geographycited by 0
  32. 2026Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Oncecited by 0
  33. 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudescited by 0
  34. 2026Do Large Language Models Encode Institutional Experience? Evidence from Cross-Linguistic Moral Reasoning Under Ambiguitycited by 0
  35. 2026Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Modelscited by 0
  36. 2026Beyond Alignment: Value Diversity as a Collective Property in Multicultural Agent Systemscited by 0
  37. 2026A Diagnostic Framework for AI Agent Behaviorcited by 0
  38. 2026When Reasoning Narrows the Move: Diversity Collapse in LLM Game Playcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thought
  2. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
  3. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior
  4. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
  5. 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
  6. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment
  7. 2026Alignment by Stereotyping: How LLMs Sacrifice Individual Distinctiveness for Cultural Adaptation
  8. 2026The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance