mlsanityData to 7 Oct 2026

Human values and AI feedback alignment

709 papers2 this weekLow confidence in this grouping: the papers may belong to more than one topic, or the group may changeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About

Keywords: alignment, human, llms, language, preferences

Key facts. Human values and AI feedback alignment: 709 papers on arXiv since 2017 (data through 7 Oct 2026). 141 of its 709 mapped papers (20%) cite “Constitutional AI: Harmlessness from AI Feedback” (2022, arXiv 2212.08073). 128 of its 709 mapped papers (18%) cite “Learning to summarize from human feedback” (2020, arXiv 2009.01325). 19 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

19 new papers, 17 Sep – 7 Oct; 2 this week.

  1. This week · 2
  2. Successive Training Stages and Large Language Model Persuasion: Effects of Misalignment, Supervised Fine-Tuning, and Preference Optimization
  3. SPEAR: Five Principles for Interactive Human-Agent Alignment
  4. Earlier · 17
  5. AI Safety via Debate is Compromised by Cognitive Biases
  6. EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans also fits: Process reward models for reasoning
  7. Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It also fits: Preference alignment for recommenders and RAG
  8. Mitigating Social Sycophancy via Pluralistic Preference Optimization
  9. Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness
  10. Demographic Pluralism: Inference-Time Modeling of Pluralistic Human Preference Distributions
  11. PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators
  12. From Normative Frameworks to Alignment Data: Constructing and Evaluating SFT and Preference Data
All 19
  1. Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks also fits: Diffusion and multimodal model alignment
  2. CRISP: Cultural Reward Modeling for Implicit Situated Propriety also fits: Reward modeling and RLHF
  3. From Constitutions to Control: Interpretable Rewards for Aligning Language Models
  4. DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
  5. Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment also fits: LLM preference optimization methods
  6. The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance
  7. When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits also fits: Preference alignment for recommenders and RAG
  8. Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
  9. Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

Foundations

Most cited by papers here

  1. Constitutional AI: Harmlessness from AI Feedback2022cited by 141 here
  2. Learning to summarize from human feedback2020cited by 128 here
  3. Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging2023cited by 56 here
  4. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference2024cited by 48 here
  5. A Roadmap to Pluralistic Alignment2024cited by 47 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Deep reinforcement learning from human preferencescited by 167 here
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 42 here
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 93 here
  4. 2020Learning to summarize from human feedbackcited by 128 here
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 92 here
  6. 2022Training language models to follow instructions with human feedbackcited by 337 here
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 165 here
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 37 here
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 60 here
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 56 here
  11. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 38 here
  12. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 42 here
12 more it builds on, not in this chain
  1. 2020Aligning AI With Shared Human Valuescited by 46 here
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 42 here
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 141 here
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 35 here
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 39 here
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 39 here
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 254 here
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 36 here
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 40 here
  10. 2024A Roadmap to Pluralistic Alignmentcited by 47 here
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 48 here
  12. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 40 here

The 24 most cited of 910.

Browse

30 of 709, ranked 6–35

  1. 6Aligning AI With Shared Human Values202046
  2. 7Personalized Language Modeling from Personalized Human Feedback202438
  3. 8The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models202435
  4. 9Discovering Language Model Behaviors with Model-Written Evaluations202235
  5. 10UltraFeedback: Boosting Language Models with Scaled AI Feedback202331
  6. 11Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration202429
  7. 12AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback202329
  8. 13Fine-tuning language models to find agreement among humans with diverse preferences202225
  9. 14LLM Evaluators Recognize and Favor Their Own Generations202423
  10. 15The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values202323
All 30
  1. 16Collective Constitutional AI: Aligning a Language Model with Public Input202422
  2. 17Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback202420
  3. 18Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision202320
  4. 19Aligning Language Models to User Opinions202319
  5. 20Prometheus: Inducing Fine-grained Evaluation Capability in Language Models202319
  6. 21Aligning to Thousands of Preferences via System Message Generalization202418
  7. 22PERSONA: A Reproducible Testbed for Pluralistic Alignment202418
  8. 23Investigating Cultural Alignment of Large Language Models202417
  9. 24Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback202317
  10. 25Training Language Models with Language Feedback at Scale202317
  11. 26Training Language Models with Language Feedback202216
  12. 27Human Feedback is not Gold Standard202315
  13. 28Cultural Bias and Cultural Alignment of Large Language Models202315
  14. 29Aligning LLMs with Individual Preferences via Interaction202414
  15. 30In conversation with Artificial Intelligence: aligning language models with human values202213
  16. 31From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models202312
  17. 32Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback202212
  18. 33Aligning LLM Agents by Learning Latent Preference from User Edits202411
  19. 34AI Alignment with Changing and Influenceable Reward Functions202411
  20. 35PersonalLLM: Tailoring LLMs to Individual Preferences202411