# Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)

arXiv: https://arxiv.org/abs/1901.00064 · HTML: https://mlsanity.com/paper/1901.00064/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2018.

## Built on by (most cited first, 22 of 22)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs](https://mlsanity.com/paper/2502.19148/) (2025)
- [Hard Choices in Artificial Intelligence](https://mlsanity.com/paper/2106.11022/) (2021)
- [`It is currently hodgepodge'': Examining AI/ML Practitioners' Challenges during Co-production of Responsible AI Values](https://mlsanity.com/paper/2307.10221/) (2023)
- [Impossibility Results in AI: A Survey](https://mlsanity.com/paper/2109.00484/) (2021)
- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Designing Fiduciary Artificial Intelligence](https://mlsanity.com/paper/2308.02435/) (2023)
- [Aligned with Whom? Direct and social goals for AI systems](https://mlsanity.com/paper/2205.04279/) (2022)
- [Value Preferences Estimation and Disambiguation in Hybrid Participatory Systems](https://mlsanity.com/paper/2402.16751/) (2024)
- [Towards Regulatable AI Systems: Technical Gaps and Policy Opportunities](https://mlsanity.com/paper/2306.12609/) (2023)
- [Towards Friendly AI: A Comprehensive Review and New Perspectives on Human-AI Alignment](https://mlsanity.com/paper/2412.15114/) (2024)
- [The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation](https://mlsanity.com/paper/2511.12804/) (2025)
- [On the Undecidability of Artificial Intelligence Alignment: Machines that Halt](https://mlsanity.com/paper/2408.08995/) (2024)
- [The Accountability Horizon: An Impossibility Theorem for Governing Human-Agent Collectives](https://mlsanity.com/paper/2604.07778/) (2026)
- [Dynamic value alignment through preference aggregation of multiple objectives](https://mlsanity.com/paper/2310.05871/) (2023)
- [Agents Need Not Know Their Purpose](https://mlsanity.com/paper/2402.09734/) (2024)
- [Relative Principals, Pluralistic Alignment, and the Structural Value Alignment Problem](https://mlsanity.com/paper/2604.20805/) (2026)
- [The Evolutionary Origin of Values: implications for AI alignment, sentience and existential risk](https://mlsanity.com/paper/2608.03361/) (2026)
- [Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment](https://mlsanity.com/paper/2609.05036/) (2026)

## Abstract (arXiv)

Utility functions or their equivalents (value functions, objective functions, loss functions, reward functions, preference orderings) are a central tool in most current machine learning systems. These mechanisms for defining goals and guiding optimization run into practical and conceptual difficulty when there are independent, multi-dimensional objectives that need to be pursued simultaneously and cannot be reduced to each other. Ethicists have proved several impossibility theorems that stem from this origin; those results appear to show that there is no way of formally specifying what it means for an outcome to be good for a population without violating strong human ethical intuitions (in such cases, the objective function is a social welfare function). We argue that this is a practical problem for any machine learning system (such as medical decision support systems or autonomous weapons) or rigidly rule-based bureaucracy that will make high stakes decisions about human lives: such systems should not use objective functions in the strict mathematical sense. We explore the alternative of using uncertain objectives, represented for instance as partially ordered preferences, or as probability distributions over total orders. We show that previously known impossibility theorems can be transformed into uncertainty theorems in both of those settings, and prove lower bounds on how much uncertainty is implied by the impossibility results. We close by proposing two conjectures about the relationship between uncertainty in objectives and severe unintended consequences from AI systems.