# Value alignment: a formal approach

arXiv: https://arxiv.org/abs/2110.09240 · HTML: https://mlsanity.com/paper/2110.09240/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 0 of its 26 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Built on by (most cited first, 18 of 18)

- [A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms](https://mlsanity.com/paper/2206.14983/) (2022)
- [A computational framework for human values](https://mlsanity.com/paper/2305.02748/) (2023)
- [Positive Alignment: Artificial Intelligence for Human Flourishing](https://mlsanity.com/paper/2605.10310/) (2026)
- [Measuring Value Alignment](https://mlsanity.com/paper/2312.15241/) (2023)
- [Predicting Privacy Preferences for Smart Devices as Norms](https://mlsanity.com/paper/2302.10650/) (2023)
- [Collaborative filtering to capture AI user's preferences as norms](https://mlsanity.com/paper/2308.02542/) (2023)
- [Algorithms for learning value-aligned policies considering admissibility relaxation](https://mlsanity.com/paper/2406.04838/) (2024)
- [Explainable Moral Values: a neuro-symbolic approach to value classification](https://mlsanity.com/paper/2410.12631/) (2024)
- [Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies](https://mlsanity.com/paper/2412.00033/) (2024)
- [The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis](https://mlsanity.com/paper/2509.10297/) (2025)
- [Instilling Organisational Values in Firefighters through Simulation-Based Training](https://mlsanity.com/paper/2512.13737/) (2025)
- [Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning](https://mlsanity.com/paper/2602.04518/) (2026)
- [From User Preferences to Base Score Extraction Functions in Gradual Argumentation (with Appendix)](https://mlsanity.com/paper/2602.14674/) (2026)
- [Human Values in Multiagent Systems](https://mlsanity.com/paper/2305.02739/) (2023)
- [Multi-Value Alignment in Normative Multi-Agent System: Evolutionary Optimisation Approach](https://mlsanity.com/paper/2305.07366/) (2023)
- [Rethinking How AI Embeds and Adapts to Human Values: Challenges and Opportunities](https://mlsanity.com/paper/2508.17104/) (2025)
- [Value-Aware Multiagent Systems](https://mlsanity.com/paper/2512.12652/) (2025)
- [Toward a Theory of Value in AI Alignment](https://mlsanity.com/paper/2608.10327/) (2026)

## Abstract (arXiv)

principles that should govern autonomous AI systems. It essentially states that a system's goals and behaviour should be aligned with human values. But how to ensure value alignment? In this paper we first provide a formal model to represent values through preferences and ways to compute value aggregations; i.e. preferences with respect to a group of agents and/or preferences with respect to sets of values. Value alignment is then defined, and computed, for a given norm with respect to a given value through the increase/decrease that it results in the preferences of future states of the world. We focus on norms as it is norms that govern behaviour, and as such, the alignment of a given system with a given value will be dictated by the norms the system follows.