mlsanityData to 14 Oct 2026

AI Alignment with Changing and Influenceable Reward Functions

arXiv 2405.17713 ↗First posted 28 May 2024cs.AI cs.LGcited by 33

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Existing AI alignment approaches assume that preferences are static, which is unrealistic: our preferences change, and may even be influenced by our interactions with AI systems themselves. To clarify the consequences of incorrectly assuming static preferences, we introduce Dynamic Reward Markov Decision Processes (DR-MDPs), which explicitly model preference changes and the AI's influence on them. We show that despite its convenience, the static-preference assumption may undermine the soundness of existing alignment techniques, leading them to implicitly reward AI systems for influencing user preferences in ways users may not truly want. We then explore potential solutions. First, we offer a unifying perspective on how an agent's optimization horizon may partially help reduce undesirable AI influence. Then, we formalize different notions of AI alignment that account for preference change from the outset. Comparing the strengths and limitations of 8 such notions of alignment, we find that they all either err towards causing undesirable AI influence, or are overly risk-averse, suggesting that a straightforward solution to the problems of changing preferences may not exist. As there is no avoiding grappling with changing preferences in real-world settings, this makes it all the more important to handle these issues with care, balancing risks and capabilities. We hope our work can provide conceptual clarity and constitute a first step towards AI alignment practices which explicitly account for (and contend with) the changing and influenceable nature of human preferences.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 176 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

60 of 176 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback16 Apr
  2. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications24 Oct
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 57

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  6. 2020Artificial Intelligence, Values and Alignmentcited by 448
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  8. 2020Performative Predictioncited by 327
  9. 2023An Overview of Catastrophic AI Riskscited by 222
  10. 2021Reinforcement learning based recommender systems: A surveycited by 188
  11. 2021Alignment of Language Agentscited by 168
  12. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
38 more
  1. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  2. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  3. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  4. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  5. 2024The Ethics of Advanced AI Assistantscited by 107
  6. 2023Eliciting Human Preferences with Language Modelscited by 92
  7. 2018Incomplete Contracting and AI Alignmentcited by 62
  8. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
  9. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 53
  10. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  11. 2018Horizon: Facebook's Open Source Applied Reinforcement Learning Platformcited by 48
  12. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  13. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45
  14. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 43
  15. 2022Performative Powercited by 43
  16. 2023Characterizing Manipulation from AI Systemscited by 43
  17. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  18. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  19. 2021Algorithmic Amplification of Politics on Twittercited by 38
  20. 2019Preferences Implicit in the State of the Worldcited by 34
  21. 2020AI Research Considerations for Human Existential Safety (ARCHES)cited by 33
  22. 2023Artificial Influence: An Analysis Of AI-Driven Persuasioncited by 33
  23. 2018Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibilitycited by 32
  24. 2023Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Mediacited by 30
  25. 2022The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimizationcited by 29
  26. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
  27. 2021Hard Choices in Artificial Intelligencecited by 26
  28. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  29. 2024The Reasons that Agents Act: Intention and Instrumental Goalscited by 23
  30. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  31. 2023On The Fragility of Learned Reward Functionscited by 21
  32. 2022Path-Specific Objectives for Safer Agent Incentivescited by 20
  33. 2023Reasoning about Causality in Gamescited by 20
  34. 2019The Assistive Multi-Armed Banditcited by 19
  35. 2022Influencing Long-Term Behavior in Multiagent Reinforcement Learningcited by 18
  36. 2024When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedbackcited by 17
  37. 2023Reinforcing User Retention in a Billion Scale Short Video Recommender Systemcited by 15
  38. 2022Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AIcited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 17; 2026: 5.

33 papers.

Sort:

By area:

  1. 2024Beyond Preferences in AI Alignmentcited by 49
  2. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 47
  3. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  4. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  5. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  6. 2025Virtual Agent Economiescited by 21
  7. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  8. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 15
  9. 2025Personalize Your LLM: Fake it then Align itcited by 15
  10. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 13
  11. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  12. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
21 more
  1. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  2. 2025LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactionscited by 8
  3. 2025LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactionscited by 7
  4. 2024SPICA: Retrieving Scenarios for Pluralistic In-Context Alignmentcited by 6
  5. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  6. 2025Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Valuecited by 6
  7. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  8. 2025Emergent Risk Awareness in Rational Agents under Resource Constraintscited by 5
  9. 2024Pluralistic Alignment Over Timecited by 3
  10. 2025Training LLM Agents to Empower Humanscited by 3
  11. 2024Aligning Generalisation Between Humans and Machinescited by 2
  12. 2025Flexible Agent Alignment with Goal Inference from Open-Ended Dialogcited by 1
  13. 2025The Burden of Interactive Alignment with Inconsistent Preferencescited by 1
  14. 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interactioncited by 1
  15. 2025Corrigibility Transformation: Constructing Goals That Accept Updatescited by 0
  16. 2025Multi-User Personalisation in Human-Robot Interaction: Resolving Preference Conflicts Using Gradual Argumentationcited by 0
  17. 2025Human-AI Interaction Alignment: Designing, Evaluating, and Evolving Value-Centered AI For Reciprocal Human-AI Futurescited by 0
  18. 2026The Boundaries of Automation: A Theory of Persistent Human Participationcited by 0
  19. 2026Co-design of LLM-based preference agents: participation may drive overtrustcited by 0
  20. 2026AI Alignment and Fiduciary Obligationcited by 0
  21. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference Learning for AI Alignment: a Causal Perspective
  2. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking
  3. 2025Direct Alignment with Heterogeneous Preferences
  4. 2024Strong Preferences Affect the Robustness of Preference Models and Value Alignment
  5. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  6. 2026Align AI to Dynamic Human-AI Workflows
  7. 2025On Monotonicity in AI Alignment
  8. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models