mlsanityData to 7 Oct 2026

RLHF and IIA: Perverse Incentives

arXiv 2312.01057 ↗First posted 2 Dec 2023cs.LG cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA). The perverse incentives induced by IIA hinder innovations on query formats and learning algorithms.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 2 Dec 2023.

Builds on

18 of 23 references.

References · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
6 more
  1. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  3. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  4. 2023A density estimation perspective on learning from pairwise human preferencescited by 23
  5. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Modelscited by 8
  6. 2022Inclusive Artificial Intelligencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  2. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  3. 2025Jackpot! Alignment as a Maximal Lotterycited by 17
  4. 2025Clone-Robust AI Alignmentcited by 7
  5. 2025The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneitycited by 1
  6. 2026Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  2. 2023Is RLHF More Difficult than Standard RL?
  3. 2023A Survey of Reinforcement Learning from Human Feedback
  4. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  6. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Models
  7. 2026Procedural Fairness Failures in RLHF from Preference Averaging
  8. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspective