mlsanityData to 7 Oct 2026

Democratizing Reward Design for Personal and Representative Value-Alignment

arXiv 2410.22203 ↗First posted 29 Oct 2024cs.AI cs.HCcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Aligning AI agents with human values is challenging due to diverse and subjective notions of values. Standard alignment methods often aggregate crowd feedback, which can result in the suppression of unique or minority preferences. We introduce Interactive-Reflective Dialogue Alignment, a method that iteratively engages users in reflecting on and specifying their subjective value definitions. This system learns individual value definitions through language-model-based preference elicitation and constructs personalized reward models that can be used to align AI behaviour. We evaluated our system through two studies with 30 participants, one focusing on "respect" and the other on ethical decision-making in autonomous vehicles. Our findings demonstrate diverse definitions of value-aligned behaviour and show that our system can accurately capture each person's unique understanding. This approach enables personalized alignment and can inform more representative and interpretable collective alignment strategies.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 85 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

14 of 85 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2020A Survey of Deep Active Learningcited by 547
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2018Conversational Recommender Systemcited by 115
  10. 2017Time Limits in Reinforcement Learningcited by 72
  11. 2017A Voting-Based System for Ethical Decision Makingcited by 55
  12. 2022Melting Pot 2.0cited by 37
2 more
  1. 2020Adapting a Kidney Exchange Algorithm to Align with Human Valuescited by 36
  2. 2016Avoiding Wireheading with Value Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Userscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Reflective Verbal Reward Design for Pluralistic Alignment
  2. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs
  3. 2024Beyond Preferences in AI Alignment
  4. 2026Toward a Theory of Value in AI Alignment
  5. 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction
  6. 2021Value alignment: a formal approach
  7. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications
  8. 2024Moral Alignment for LLM Agents