mlsanityData to 14 Oct 2026

A density estimation perspective on learning from pairwise human preferences

arXiv 2311.14115 ↗First posted 23 Nov 2023cs.LG cs.AI cs.CLcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from human feedback (LHF) -- and in particular learning from pairwise preferences -- has recently become a crucial ingredient in training large language models (LLMs), and has been the subject of much research. Most recent works frame it as a reinforcement learning problem, where a reward function is learned from pairwise preference data and the LLM is treated as a policy which is adapted to maximize the rewards, often under additional regularization constraints. We propose an alternative interpretation which centers on the generative process for pairwise preferences and treats LHF as a density estimation problem. We provide theoretical and empirical results showing that for a family of generative processes defined via preference behavior distribution equations, training a reward function on pairwise preferences effectively models an annotator's implicit preference distribution. Finally, we discuss and present findings on "annotator misspecification" -- failure cases where wrong modeling assumptions are made about annotator behavior, resulting in poorly-adapted models -- suggesting that approaches that learn from pairwise human preferences could have trouble learning from a population of annotators with diverse viewpoints.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 35 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

22 of 35 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models19 May
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  5. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  6. 2023Chain of Hindsight Aligns Language Models with Feedback6 Feb
  7. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  10. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  7. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  9. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  10. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 82
  11. 2018Capturing human category representations by sampling in deep feature spacescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 12; 2026: 1.

23 papers.

Sort:

By area:

  1. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  2. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  3. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  4. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
  5. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  6. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  7. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 19
  8. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  9. 2024Optimizing Language Models for Human Preferences is a Causal Inference Problemcited by 8
  10. 2024On Extending Direct Preference Optimization to Accommodate Tiescited by 8
  11. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  12. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 8
11 more
  1. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  2. 2023RLHF and IIA: Perverse Incentivescited by 6
  3. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  4. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  5. 2025Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curationcited by 3
  6. 2025PIPA: Preference Alignment as Prior-Informed Statistical Estimationcited by 1
  7. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
  8. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  9. 2025Score-Based Density Estimation from Pairwise Comparisonscited by 0
  10. 2025Buzz, Choose, Forget: A Meta-Bandit Framework for Bee-Like Decision Makingcited by 0
  11. 2026Pairwise Reference Alignment as a Model-Level Ordinal Observablecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  4. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
  5. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  6. 2026A Markov Chain Approach to Preference Alignment
  7. 2025Learning a Canonical Basis of Human Preferences from Binary Ratings
  8. 2024Towards Comprehensive Preference Data Collection for Reward Modeling