# What are human values, and how do we align AI to them?

arXiv: https://arxiv.org/abs/2404.10636 · HTML: https://mlsanity.com/paper/2404.10636/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [Bridging Systems: Open Problems for Countering Destructive Divisiveness across Ranking, Recommenders, and Governance](https://mlsanity.com/paper/2301.09976/) (2023)

## Built on by (most cited first, 30 of 32)

- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [Value Profiles for Encoding Human Variation](https://mlsanity.com/paper/2503.15484/) (2025)
- [Strong and weak alignment of large language models with human values](https://mlsanity.com/paper/2408.04655/) (2024)
- [Evaluating the Prompt Steerability of Large Language Models](https://mlsanity.com/paper/2411.12405/) (2024)
- [AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents](https://mlsanity.com/paper/2506.04018/) (2025)
- [AI Alignment at Your Discretion](https://mlsanity.com/paper/2502.10441/) (2025)
- [The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models](https://mlsanity.com/paper/2406.11096/) (2024)
- [Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models](https://mlsanity.com/paper/2409.12106/) (2024)
- [Generative Value Conflicts Reveal LLM Priorities](https://mlsanity.com/paper/2509.25369/) (2025)
- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Value Drifts: Tracing Value Alignment During LLM Post-Training](https://mlsanity.com/paper/2510.26707/) (2025)
- [Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions](https://mlsanity.com/paper/2503.00940/) (2025)
- [Representative Social Choice: From Learning Theory to AI Alignment](https://mlsanity.com/paper/2410.23953/) (2024)
- [Societal Alignment Frameworks Can Improve LLM Alignment](https://mlsanity.com/paper/2503.00069/) (2025)
- [Resource Rational Contractualism Should Guide AI Alignment](https://mlsanity.com/paper/2506.17434/) (2025)
- [PORT: Preference Optimization on Reasoning Traces](https://mlsanity.com/paper/2406.16061/) (2024)
- [SPICA: Retrieving Scenarios for Pluralistic In-Context Alignment](https://mlsanity.com/paper/2411.10912/) (2024)
- [Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks](https://mlsanity.com/paper/2504.01850/) (2025)
- [Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value](https://mlsanity.com/paper/2512.03399/) (2025)
- [Analyzing the Ethical Logic of Eight Large Language Models](https://mlsanity.com/paper/2501.08951/) (2025)
- [AI-Enhanced Deliberative Democracy and the Future of the Collective Will](https://mlsanity.com/paper/2503.05830/) (2025)
- [Value Alignment of Social Media Ranking Algorithms](https://mlsanity.com/paper/2509.14434/) (2025)
- [Human Values Matter: Investigating How Misalignment Shapes Collective Behaviors in LLM Agent Communities](https://mlsanity.com/paper/2604.05339/) (2026)
- [VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models](https://mlsanity.com/paper/2602.03160/) (2026)
- [Large Language Models Develop Novel Social Biases Through Adaptive Exploration](https://mlsanity.com/paper/2511.06148/) (2025)
- [Bringing AI Participation Down to Scale: A Comment on Open AIs Democratic Inputs to AI Project](https://mlsanity.com/paper/2407.11613/) (2024)
- [Intuitions of Compromise: Utilitarianism vs. Contractualism](https://mlsanity.com/paper/2410.05496/) (2024)
- [Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?](https://mlsanity.com/paper/2505.13257/) (2025)
- [Alignment Is Not Enough: A Relational Framework for Moral Standing in Human-AI Interaction](https://mlsanity.com/paper/2603.00078/) (2026)
- [Can "AI" Be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMs](https://mlsanity.com/paper/2604.20791/) (2026)

## Abstract (arXiv)

There is an emerging consensus that we need to align AI systems with human values (Gabriel, 2020; Ji et al., 2024), but it remains unclear how to apply this to language models in practice. We split the problem of "aligning to human values" into three parts: first, eliciting values from people; second, reconciling those values into an alignment target for training ML models; and third, actually training the model. In this paper, we focus on the first two parts, and ask the question: what are "good" ways to synthesize diverse human inputs about values into a target for aligning language models? To answer this question, we first define a set of 6 criteria that we believe must be satisfied for an alignment target to shape model behavior in accordance with human values. We then propose a process for eliciting and reconciling values called Moral Graph Elicitation (MGE), which uses a large language model to interview participants about their values in particular contexts; our approach is inspired by the philosophy of values advanced by Taylor (1977), Chang (2004), and others. We trial MGE with a representative sample of 500 Americans, on 3 intentionally divisive prompts (e.g. advice about abortion). Our results demonstrate that MGE is promising for improving model alignment across all 6 criteria. For example, almost all participants (89.1%) felt well represented by the process, and (89%) thought the final moral graph was fair, even if their value wasn't voted as the wisest. Our process often results in "expert" values (e.g. values from women who have solicited abortion advice) rising to the top of the moral graph, without defining who is considered an expert in advance.