# Human values and AI feedback alignment

Area: [LLM alignment / RLHF / preference optimisation](https://mlsanity.com/alignment/index.md). HTML: https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/

## Key facts

- Human values and AI feedback alignment: 709 papers on arXiv since 2017 (data through 7 Oct 2026).
- 141 of its 709 mapped papers (20%) cite “Constitutional AI: Harmlessness from AI Feedback” (2022, arXiv 2212.08073).
- 128 of its 709 mapped papers (18%) cite “Learning to summarize from human feedback” (2020, arXiv 2009.01325).
- 19 new papers 17 Sep–7 Oct.

## Most cited here

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022), cited by 141 papers here
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020), cited by 128 papers here
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023), cited by 56 papers here
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024), cited by 48 papers here
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024), cited by 47 papers here
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020), cited by 46 papers here
- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024), cited by 38 papers here
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022), cited by 35 papers here
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024), cited by 35 papers here
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023), cited by 31 papers here

## New (17 Sep–7 Oct)

- [Successive Training Stages and Large Language Model Persuasion: Effects of Misalignment, Supervised Fine-Tuning, and Preference Optimization](https://mlsanity.com/paper/2610.09964/) (2026-10-07)
- [SPEAR: Five Principles for Interactive Human-Agent Alignment](https://mlsanity.com/paper/2610.07204/) (2026-10-05)
- [AI Safety via Debate is Compromised by Cognitive Biases](https://mlsanity.com/paper/2610.05461/) (2026-10-04)
- [EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans](https://mlsanity.com/paper/2610.05370/) (2026-10-04)
- [Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It](https://mlsanity.com/paper/2610.03195/) (2026-10-02)
- [Mitigating Social Sycophancy via Pluralistic Preference Optimization](https://mlsanity.com/paper/2610.02568/) (2026-10-01)
- [Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness](https://mlsanity.com/paper/2610.00568/) (2026-09-30)
- [Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributions](https://mlsanity.com/paper/2609.38555/) (2026-09-29)
- [PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators](https://mlsanity.com/paper/2609.36086/) (2026-09-28)
- [From Normative Frameworks to Alignment Data: Constructing and Evaluating SFT and Preference Data](https://mlsanity.com/paper/2609.35201/) (2026-09-28)
- [Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks](https://mlsanity.com/paper/2609.34506/) (2026-09-28)
- [CRISP: Cultural Reward Modeling for Implicit Situated Propriety](https://mlsanity.com/paper/2609.34345/) (2026-09-28)
- [From Constitutions to Control: Interpretable Rewards for Aligning Language Models](https://mlsanity.com/paper/2609.33086/) (2026-09-27)
- [DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration](https://mlsanity.com/paper/2609.31215/) (2026-09-25)
- [Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment](https://mlsanity.com/paper/2609.26929/) (2026-09-22)
- [The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance](https://mlsanity.com/paper/2609.25760/) (2026-09-22)
- [When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits](https://mlsanity.com/paper/2609.24194/) (2026-09-21)
- [Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment](https://mlsanity.com/paper/2609.23640/) (2026-09-20)
- [Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models](https://mlsanity.com/paper/2609.21094/) (2026-09-17)