mlsanityData to 7 Oct 2026

Benchmarking Overton Pluralism in LLMs

arXiv 2512.01351 ↗First posted 1 Dec 2025cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce OVERTONBENCH, a novel framework for measuring Overton pluralism in LLMs--the extent to which diverse viewpoints are represented in model outputs. We (i) formalize Overton pluralism as a set coverage metric (OVERTONSCORE), (ii) conduct a large-scale U.S.-representative human study (N = 1208; 60 questions; 8 LLMs), and (iii) develop an automated benchmark that closely reproduces human judgments. On average, models achieve OVERTONSCOREs of 0.35--0.41, with DeepSeek V3 performing best; yet all models remain far below the theoretical maximum of 1.0, revealing substantial headroom for improvement. Because repeated large-scale human studies are costly and slow, scalable evaluation tools are essential for model development. Hence, we propose an automated benchmark that achieves high rank correlation with human judgments (ρ = 0.88), providing a practical proxy without replacing human assessment. By turning pluralistic alignment from a normative aim into a measurable benchmark, our work establishes a foundation for systematic progress toward more pluralistic LLMs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

29 of 43 references.

References · 29

  1. 2023Whose Opinions Do Language Models Reflect?cited by 759
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  4. 2020Artificial Intelligence, Values and Alignmentcited by 443
  5. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  6. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  7. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  8. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  9. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  10. 2024Investigating Cultural Alignment of Large Language Modelscited by 140
  11. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  12. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
17 more
  1. 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 120
  2. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  3. 2020Participation is not a Design Fix for Machine Learningcited by 110
  4. 2021Algorithmic Monoculture and Social Welfarecited by 98
  5. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  6. 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 89
  7. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  8. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  9. 2023Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Modelscited by 42
  10. 2023Opportunities and Risks of LLMs for Scalable Deliberation with Poliscited by 41
  11. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  12. 2024On the Relationship between Truth and Political Bias in Language Modelscited by 31
  13. 2025Value Profiles for Encoding Human Variationcited by 29
  14. 2023Moral Machine or Tyranny of the Majority?cited by 22
  15. 2024Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigationcited by 17
  16. 2025VITAL: A New Dataset for Benchmarking Pluralistic Alignment in Healthcarecited by 17
  17. 2024Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026AI Alignment Breaks at the Edgecited by 3
  2. 2026Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contextscited by 2
  3. 2026What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Modelscited by 2
  4. 2026Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBenchcited by 1
  5. 2026AI Alignment From Social Choice Perspectivescited by 0
  6. 2026A Roadmap to Impactful Pluralistic Alignment Researchcited by 0
  7. 2026Where Models Converge and Humans Diverge: A Coverage Framework for Distributional Pluralism in Open-Ended Generationcited by 0