# Relative Principals, Pluralistic Alignment, and the Structural Value Alignment Problem

arXiv: https://arxiv.org/abs/2604.20805 · HTML: https://mlsanity.com/paper/2604.20805/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 73 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 30 of 30)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Runaway Feedback Loops in Predictive Policing](https://mlsanity.com/paper/1706.09847/) (2017)
- [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://mlsanity.com/paper/2309.00779/) (2023)
- [Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing](https://mlsanity.com/paper/2001.00964/) (2020)
- [DICES Dataset: Diversity in Conversational AI Evaluation for Safety](https://mlsanity.com/paper/2306.11247/) (2023)
- [Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?](https://mlsanity.com/paper/2502.15657/) (2025)
- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)
- [Pragmatic-Pedagogic Value Alignment](https://mlsanity.com/paper/1707.06354/) (2017)
- [TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI](https://mlsanity.com/paper/2306.06924/) (2023)
- [Value-laden Disciplinary Shifts in Machine Learning](https://mlsanity.com/paper/1912.01172/) (2019)
- [Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)](https://mlsanity.com/paper/1901.00064/) (2018)
- [Concrete Problems in AI Safety, Revisited](https://mlsanity.com/paper/2401.10899/) (2023)
- [AI Art is Theft: Labour, Extraction, and Exploitation, Or, On the Dangers of Stochastic Pollocks](https://mlsanity.com/paper/2401.06178/) (2024)
- [Metaethical Perspectives on 'Benchmarking' AI Ethics](https://mlsanity.com/paper/2204.05151/) (2022)
- [The Value of Disagreement in AI Design, Evaluation, and Alignment](https://mlsanity.com/paper/2505.07772/) (2025)
- [Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?](https://mlsanity.com/paper/2109.08131/) (2021)
- [The Partially Observable Off-Switch Game](https://mlsanity.com/paper/2411.17749/) (2024)
- [Observation Interference in Partially Observable Assistance Games](https://mlsanity.com/paper/2412.17797/) (2024)

## Abstract (arXiv)

The value alignment problem for artificial intelligence (AI) is often framed as a purely technical or normative challenge, sometimes focused on hypothetical future systems. I argue that the problem is better understood as a structural question about governance: not whether an AI system is aligned in the abstract, but whether it is aligned enough, for whom, and at what cost. Drawing on the principal-agent framework from economics, this paper reconceptualises misalignment as arising along three interacting axes: objectives, information, and principals. The three-axis framework provides a systematic way of diagnosing why misalignment arises in real-world systems and clarifies that alignment cannot be treated as a single technical property of models but an outcome shaped by how objectives are specified, how information is distributed, and whose interests count in practice. The core contribution of this paper is to show that the three-axis decomposition implies that alignment is fundamentally a problem of governance rather than engineering alone. From this perspective, alignment is inherently pluralistic and context-dependent, and resolving misalignment involves trade-offs among competing values. Because misalignment can occur along each axis -- and affect stakeholders differently -- the structural description shows that alignment cannot be "solved" through technical design alone, but must be managed through ongoing institutional processes that determine how objectives are set, how systems are evaluated, and how affected communities can contest or reshape those decisions.