mlsanityData to 13 Oct 2026

Artificial Intelligence, Values and Alignment

arXiv 2001.09768 ↗First posted 13 Jan 2020cs.CYcited by 443

Not on the map. Why?

Cited by 104 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper looks at philosophical questions that arise in the context of AI alignment. It defends three propositions. First, normative and technical aspects of the AI alignment problem are interrelated, creating space for productive engagement between people working in both domains. Second, it is important to be clear about the goal of alignment. There are significant differences between AI that aligns with instructions, intentions, revealed preferences, ideal preferences, interests and values. A principle-based approach to AI alignment, which combines these elements in a systematic way, has considerable advantages in this context. Third, the central challenge for theorists is not to identify 'true' moral principles for AI; rather, it is to identify fair principles for alignment, that receive reflective endorsement despite widespread variation in people's moral beliefs. The final part of the paper explores three ways in which fair principles for AI alignment could potentially be identified.

Key facts. Cited by 443 papers in the map (data through 13 Oct 2026). First posted on arXiv 13 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 17; 2022: 31; 2023: 67; 2024: 98; 2025: 135; 2026: 95.

443 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,365
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 717
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 648
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 464
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 428
  7. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  8. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  9. 2021Unsolved Problems in ML Safetycited by 291
  10. 2022The Alignment Problem from a Deep Learning Perspectivecited by 252
  11. 2022Defining and Characterizing Reward Hackingcited by 198
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
38 more
  1. 2024A Roadmap to Pluralistic Alignmentcited by 179
  2. 2021Alignment of Language Agentscited by 168
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 152
  4. 2024Continual Learning for Large Language Models: A Surveycited by 144
  5. 2023Getting aligned on representational alignmentcited by 142
  6. 2021Collaborating with Humans without Human Datacited by 140
  7. 2023Auditing large language models: a three-layered approachcited by 136
  8. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 132
  9. 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 123
  10. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  11. 2023Foundation Models and Fair Usecited by 116
  12. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  13. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  14. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  15. 2024Systematic Biases in LLM Simulations of Debatescited by 99
  16. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
  17. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  18. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 78
  19. 2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 62
  20. 2023Explaining black box text modules in natural language with language modelscited by 62
  21. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
  22. 2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 55
  23. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  24. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  25. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  26. 2025Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?cited by 51
  27. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 50
  28. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  29. 2021The Who in XAI: How AI Background Shapes Perceptions of AI Explanationscited by 48
  30. 2024Beyond Preferences in AI Alignmentcited by 47
  31. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  32. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  33. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  34. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  35. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 44
  36. 2023Exploring the psychology of LLMs' Moral and Legal Reasoningcited by 43
  37. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  38. 2023Flames: Benchmarking Value Alignment of LLMs in Chinesecited by 37

50 most cited of 443 All 443 (JSON)