# Artificial Intelligence, Values and Alignment

arXiv: https://arxiv.org/abs/2001.09768 · HTML: https://mlsanity.com/paper/2001.09768/

## Key facts

- Cited by 448 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2020.

## Built on by (most cited first, 30 of 448)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [RL with KL penalties is better viewed as Bayesian inference](https://mlsanity.com/paper/2205.11275/) (2022)
- [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024)
- [Black-Box Prompt Optimization: Aligning Large Language Models without Model Training](https://mlsanity.com/paper/2311.04155/) (2023)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [Identifying and Mitigating the Security Risks of Generative AI](https://mlsanity.com/paper/2308.14840/) (2023)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [Training Socially Aligned Language Models on Simulated Social Interactions](https://mlsanity.com/paper/2305.16960/) (2023)

## Abstract (arXiv)

This paper looks at philosophical questions that arise in the context of AI alignment. It defends three propositions. First, normative and technical aspects of the AI alignment problem are interrelated, creating space for productive engagement between people working in both domains. Second, it is important to be clear about the goal of alignment. There are significant differences between AI that aligns with instructions, intentions, revealed preferences, ideal preferences, interests and values. A principle-based approach to AI alignment, which combines these elements in a systematic way, has considerable advantages in this context. Third, the central challenge for theorists is not to identify 'true' moral principles for AI; rather, it is to identify fair principles for alignment, that receive reflective endorsement despite widespread variation in people's moral beliefs. The final part of the paper explores three ways in which fair principles for AI alignment could potentially be identified.