# Generalizing while preserving monotonicity in comparison-based preference learning models

arXiv: https://arxiv.org/abs/2506.08616 · HTML: https://mlsanity.com/paper/2506.08616/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024)
- [Generalized Preference Optimization: A Unified Approach to Offline Alignment](https://mlsanity.com/paper/2402.05749/) (2024)
- [Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization](https://mlsanity.com/paper/2410.08847/) (2024)
- [A Voting-Based System for Ethical Decision Making](https://mlsanity.com/paper/1709.06692/) (2017)
- [Preference Learning Algorithms Do Not Learn Preference Rankings](https://mlsanity.com/paper/2405.19534/) (2024)
- [Tournesol: A quest for a large, secure and trustworthy database of reliable human judgments](https://mlsanity.com/paper/2107.07334/) (2021)
- [Generalized Bradley-Terry Models for Score Estimation from Paired Comparisons](https://mlsanity.com/paper/2308.08644/) (2023)
- [A Generalized Model for Multidimensional Intransitivity](https://mlsanity.com/paper/2409.19325/) (2024)
- [A User Study of Perceived Carbon Footprint](https://mlsanity.com/paper/1911.11658/) (2019)

## Built on by (most cited first, 1 of 1)

- [humancompatible.detect: a Python Toolkit for Detecting Bias in AI Models](https://mlsanity.com/paper/2509.24340/) (2025)

## Abstract (arXiv)

If you tell a learning model that you prefer an alternative a over another alternative b, then you probably expect the model to be monotone, that is, the valuation of a increases, and that of b decreases. Yet, perhaps surprisingly, many widely deployed comparison-based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley-Terry models. Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that are monotone. Namely, we propose a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantee monotonicity. Our experiments show that this monotonicity is far from being a general guarantee, and that our new class of generalizing models improves accuracy, especially when the dataset is limited.