# The Value of Disagreement in AI Design, Evaluation, and Alignment

arXiv: https://arxiv.org/abs/2505.07772 · HTML: https://mlsanity.com/paper/2505.07772/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2025.

## Built on by (most cited first, 12 of 12)

- [Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset](https://mlsanity.com/paper/2507.09650/) (2025)
- [Culture is Everywhere: A Call for Intentionally Cultural Evaluation](https://mlsanity.com/paper/2509.01301/) (2025)
- [Beyond "Hallucinations": A Framework for Stable Human-AI Reasoning](https://mlsanity.com/paper/2510.14665/) (2025)
- [Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks](https://mlsanity.com/paper/2603.05801/) (2026)
- [Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scale](https://mlsanity.com/paper/2601.18730/) (2026)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations](https://mlsanity.com/paper/2509.25492/) (2025)
- [RLHF May Not Reflect Genuine Preferences](https://mlsanity.com/paper/2604.03238/) (2026)
- [Is your AI Model Accurate Enough? The Difficult Choices Behind Rigorous AI Development and the EU AI Act](https://mlsanity.com/paper/2604.03254/) (2026)
- [Relative Principals, Pluralistic Alignment, and the Structural Value Alignment Problem](https://mlsanity.com/paper/2604.20805/) (2026)
- [On Epistemic Diversity in Large Language Models](https://mlsanity.com/paper/2609.04835/) (2026)
- [Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment](https://mlsanity.com/paper/2609.05036/) (2026)

## Abstract (arXiv)

Disagreements are widespread across the design, evaluation, and alignment pipelines of artificial intelligence (AI) systems. Yet, standard practices in AI development often obscure or eliminate disagreement, resulting in an engineered homogenization that can be epistemically and ethically harmful, particularly for marginalized groups. In this paper, we characterize this risk, and develop a normative framework to guide practical reasoning about disagreement in the AI lifecycle. Our contributions are two-fold. First, we introduce the notion of perspectival homogenization, characterizing it as a coupled ethical-epistemic risk that arises when an aspect of an AI system's development unjustifiably suppresses disagreement and diversity of perspectives. We argue that perspectival homogenization is best understood as a procedural risk, which calls for targeted interventions throughout the AI development pipeline. Second, we propose a normative framework to guide such interventions, grounded in lines of research that explain why disagreement can be epistemically beneficial, and how its benefits can be realized in practice. We apply this framework to key design questions across three stages of AI development tasks: when disagreement is epistemically valuable; whose perspectives should be included and preserved; how to structure tasks and navigate trade-offs; and how disagreement should be documented and communicated. In doing so, we challenge common assumptions in AI practice, offer a principled foundation for emerging participatory and pluralistic approaches, and identify actionable pathways for future work in AI design and governance.