# Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback

arXiv: https://arxiv.org/abs/2511.10032 · HTML: https://mlsanity.com/paper/2511.10032/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 18 of its 61 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 18 of 18)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [A Voting-Based System for Ethical Decision Making](https://mlsanity.com/paper/1709.06692/) (2017)
- [Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives](https://mlsanity.com/paper/2411.04991/) (2024)
- [Moral Alignment for LLM Agents](https://mlsanity.com/paper/2410.01639/) (2024)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization](https://mlsanity.com/paper/2409.03650/) (2024)
- [Interactive AI Alignment: Specification, Process, and Evaluation Alignment](https://mlsanity.com/paper/2311.00710/) (2023)
- [Direct Alignment with Heterogeneous Preferences](https://mlsanity.com/paper/2502.16320/) (2025)
- [Learning When to Advise Human Decision Makers](https://mlsanity.com/paper/2209.13578/) (2022)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift](https://mlsanity.com/paper/2407.18676/) (2024)
- [Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions](https://mlsanity.com/paper/2503.00940/) (2025)
- [An Evaluation of GPT-4 on the ETHICS Dataset](https://mlsanity.com/paper/2309.10492/) (2023)
- [Who Gets the Kidney? Human-AI Alignment, Indecision, and Moral Values](https://mlsanity.com/paper/2506.00079/) (2025)

## Built on by (most cited first, 6 of 6)

- [The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions](https://mlsanity.com/paper/2608.05583/) (2026)
- [The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisions](https://mlsanity.com/paper/2605.12717/) (2026)
- [Unsupervised Elicitation of Moral Values from Language Models](https://mlsanity.com/paper/2601.17728/) (2026)
- [Internal Pluralism and the Limits of Pairwise Comparisons](https://mlsanity.com/paper/2607.02672/) (2026)
- [Position: We Need Practical AI Alignment Methods to Mirror Human Reasoning](https://mlsanity.com/paper/2608.12372/) (2026)
- [Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture](https://mlsanity.com/paper/2608.14566/) (2026)

## Abstract (arXiv)

Alignment methods in moral domains seek to elicit moral preferences of human stakeholders and incorporate them into AI. This presupposes moral preferences as static targets, but such preferences often evolve over time. Proper alignment of AI to dynamic human preferences should ideally account for "legitimate" changes to moral reasoning, while ignoring changes related to attention deficits, cognitive biases, or other arbitrary factors. However, common AI alignment approaches largely neglect temporal changes in preferences, posing serious challenges to proper alignment, especially in high-stakes applications of AI, e.g., in healthcare domains, where misalignment can jeopardize the trustworthiness of the system and yield serious individual and societal harms. This work investigates the extent to which people's moral preferences change over time, and the impact of such changes on AI alignment. Our study is grounded in the kidney allocation domain, where we elicit responses to pairwise comparisons of hypothetical kidney transplant patients from over 400 participants across 3-5 sessions. We find that, on average, participants change their response to the same scenario presented at different times around 6-20% of the time (exhibiting "response instability"). Additionally, we observe significant shifts in several participants' retrofitted decision-making models over time (capturing "model instability"). The predictive performance of simple AI models decreases as a function of both response and model instability. Moreover, predictive performance diminishes over time, highlighting the importance of accounting for temporal changes in preferences during training. These findings raise fundamental normative and technical challenges relevant to AI alignment, highlighting the need to better understand the object of alignment (what to align to) when user preferences change significantly over time.