mlsanityData to 14 Oct 2026

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift

arXiv 2407.18676 ↗First posted 26 Jul 2024cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Current Large Language Model (LLM) preference optimization algorithms do not account for temporal preference drift, which can lead to severe misalignment. To address this limitation, we propose Non-Stationary Direct Preference Optimisation (NS-DPO) that models time-dependent reward functions with a Dynamic Bradley-Terry model. NS-DPO proposes a computationally efficient solution by introducing only a single discount parameter in the loss function, which is used for exponential weighting that proportionally focuses learning on more time-relevant datapoints. We theoretically analyze the convergence of NS-DPO in a general setting where the exact nature of the preference drift is not known, providing upper bounds on the estimation error and regret caused by non-stationary preferences. Finally, we demonstrate the effectiveness of NS-DPO for fine-tuning LLMs under drifting preferences. Using scenarios where various levels of preference drift is introduced, with popular LLM reward models and datasets, we show that NS-DPO fine-tuned LLMs remain robust under non-stationarity, significantly outperforming baseline algorithms that ignore temporal preference changes, without sacrificing performance in stationary cases.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 79 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

59 of 79 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  2. 2024PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning25 Jun
  3. 2024Discovering Preference Optimization Algorithms with and for Large Language Models12 Jun
  4. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing8 Jun
  5. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process20 May
  8. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  9. 2024Direct Preference Optimization with an Offset16 Feb
  10. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  11. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  12. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
3 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 44

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2024Self-Rewarding Language Modelscited by 681
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
32 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  7. 2024Iterative Reasoning Preference Optimizationcited by 246
  8. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  9. 2023Nash Learning from Human Feedbackcited by 224
  10. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  11. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  12. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  13. 2020Improved Optimistic Algorithms for Logistic Banditscited by 118
  14. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  15. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  16. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  17. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  18. 2019Weighted Linear Bandits for Non-Stationary Environmentscited by 86
  19. 2018Learning to Optimize under Non-Stationaritycited by 85
  20. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  21. 2022X-Risk Analysis for AI Researchcited by 60
  22. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  23. 2021A Simple Approach for Non-stationary Linear Banditscited by 54
  24. 2016Time-Varying Gaussian Process Bandit Optimizationcited by 50
  25. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  26. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  27. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  28. 2022Offline Reinforcement Learning with Differential Privacycited by 20
  29. 2021Regret Bounds for Generalized Linear Bandits under Parameter Driftcited by 13
  30. 2020Nonparametric Estimation in the Dynamic Bradley-Terry Modelcited by 8
  31. 2018Modelling and Analysis of Temporal Preference Drifts Using A Component-Based Factorised Latent Approachcited by 3
  32. 2023A Spectral Approach for the Dynamic Bradley-Terry Modelcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 36
  2. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  3. 2025Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoningcited by 12
  4. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  5. 2025Orchestrating Human-AI Teams: The Manager Agent as a Unifying Research Challengecited by 8
  6. 2025Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedbackcited by 6
  7. 2025ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignmentcited by 3
  8. 2026Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judgecited by 2
  9. 2026Autoregressive Direct Preference Optimizationcited by 1
  10. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1
  11. 2026Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferencescited by 0
  12. 2026Meta-Learning Preferences for Multilingual LLM Alignmentcited by 0
1 more
  1. 2026Benchmarking Psychological Dynamics in Generative Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
  2. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  3. 2024Multi-Reference Preference Optimization for Large Language Models
  4. 2026LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
  5. 2025Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
  6. 2025KL Penalty Control via Perturbation for Direct Preference Optimization
  7. 2024Minor DPO reject penalty to increase training robustness
  8. 2025Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift