mlsanityData to 14 Oct 2026

On Monotonicity in AI Alignment

arXiv 2506.08998 ↗First posted 10 Jun 2025math.ST cs.LG stat.MLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Comparison-based preference learning has become central to the alignment of AI models with human preferences. However, these methods may behave counterintuitively. After empirically observing that, when accounting for a preference for response y over z, the model may actually decrease the probability (and reward) of generating y (an observation also made by others), this paper investigates the root causes of (non) monotonicity, for a general comparison-based preference learning framework that subsumes Direct Preference Optimization (DPO), Generalized Preference Optimization (GPO) and Generalized Bradley-Terry (GBT). Under mild assumptions, we prove that such methods still satisfy what we call local pairwise monotonicity. We also provide a bouquet of formalizations of monotonicity, and identify sufficient conditions for their guarantee, thereby providing a toolbox to evaluate how prone learning models are to monotonicity violations. These results clarify the limitations of current methods and provide guidance for developing more trustworthy preference learning algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 46 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

27 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  2. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  3. 2024Discovering Preference Optimization Algorithms with and for Large Language Models12 Jun
  4. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  7. 2024Iterative Reasoning Preference Optimizationcited by 246
  8. 2023Rich Human Feedback for Text-to-Image Generationcited by 141
  9. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  10. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  11. 2018Incomplete Contracting and AI Alignmentcited by 62
  12. 2017A Voting-Based System for Ethical Decision Makingcited by 55
7 more
  1. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 47
  2. 2024SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Datasetcited by 35
  3. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  4. 2021Tournesol: A quest for a large, secure and trustworthy database of reliable human judgmentscited by 11
  5. 2023Generalized Bradley-Terry Models for Score Estimation from Paired Comparisonscited by 9
  6. 2019A User Study of Perceived Carbon Footprintcited by 2
  7. 2024Enhancing Image Caption Generation Using Reinforcement Learning with Human Feedbackcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  2. 2024AI Alignment with Changing and Influenceable Reward Functions
  3. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
  4. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
  5. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  6. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  7. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  8. 2025Generalizing while preserving monotonicity in comparison-based preference learning models