# Benchmarking Overton Pluralism in LLMs

arXiv: https://arxiv.org/abs/2512.01351 · HTML: https://mlsanity.com/paper/2512.01351/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 29 of 29)

- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [MaxMin-RLHF: Alignment with Diverse Human Preferences](https://mlsanity.com/paper/2402.08925/) (2024)
- [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [Participation is not a Design Fix for Machine Learning](https://mlsanity.com/paper/2007.02423/) (2020)
- [Algorithmic Monoculture and Social Welfare](https://mlsanity.com/paper/2101.05853/) (2021)
- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters](https://mlsanity.com/paper/2410.24190/) (2024)
- [Unintended Impacts of LLM Alignment on Global Representation](https://mlsanity.com/paper/2402.15018/) (2024)
- [Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models](https://mlsanity.com/paper/2310.12481/) (2023)
- [Opportunities and Risks of LLMs for Scalable Deliberation with Polis](https://mlsanity.com/paper/2306.11932/) (2023)
- [From Distributional to Overton Pluralism: Investigating Large Language Model Alignment](https://mlsanity.com/paper/2406.17692/) (2024)
- [On the Relationship between Truth and Political Bias in Language Models](https://mlsanity.com/paper/2409.05283/) (2024)
- [Value Profiles for Encoding Human Variation](https://mlsanity.com/paper/2503.15484/) (2025)
- [Moral Machine or Tyranny of the Majority?](https://mlsanity.com/paper/2305.17319/) (2023)
- [Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation](https://mlsanity.com/paper/2402.01705/) (2024)
- [VITAL: A New Dataset for Benchmarking Pluralistic Alignment in Healthcare](https://mlsanity.com/paper/2502.13775/) (2025)
- [Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models](https://mlsanity.com/paper/2412.16746/) (2024)

## Built on by (most cited first, 7 of 7)

- [Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts](https://mlsanity.com/paper/2602.03545/) (2026)
- [AI Alignment Breaks at the Edge](https://mlsanity.com/paper/2602.20042/) (2026)
- [What Does the AI Doctor Value? Auditing Pluralism in the Clinical Ethics of Language Models](https://mlsanity.com/paper/2605.18738/) (2026)
- [AI Alignment From Social Choice Perspectives](https://mlsanity.com/paper/2606.21550/) (2026)
- [Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench](https://mlsanity.com/paper/2605.17079/) (2026)
- [A Roadmap to Impactful Pluralistic Alignment Research](https://mlsanity.com/paper/2607.22305/) (2026)
- [Where Models Converge and Humans Diverge: A Coverage Framework for Distributional Pluralism in Open-Ended Generation](https://mlsanity.com/paper/2608.05576/) (2026)

## Abstract (arXiv)

We introduce OVERTONBENCH, a novel framework for measuring Overton pluralism in LLMs--the extent to which diverse viewpoints are represented in model outputs. We (i) formalize Overton pluralism as a set coverage metric (OVERTONSCORE), (ii) conduct a large-scale U.S.-representative human study (N = 1208; 60 questions; 8 LLMs), and (iii) develop an automated benchmark that closely reproduces human judgments. On average, models achieve OVERTONSCOREs of 0.35--0.41, with DeepSeek V3 performing best; yet all models remain far below the theoretical maximum of 1.0, revealing substantial headroom for improvement. Because repeated large-scale human studies are costly and slow, scalable evaluation tools are essential for model development. Hence, we propose an automated benchmark that achieves high rank correlation with human judgments (ρ = 0.88), providing a practical proxy without replacing human assessment. By turning pluralistic alignment from a normative aim into a measurable benchmark, our work establishes a foundation for systematic progress toward more pluralistic LLMs.