mlsanityData to 14 Oct 2026

Strategyproof Reinforcement Learning from Human Feedback

arXiv 2503.09561 ↗First posted 12 Mar 2025cs.LGcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

We study Reinforcement Learning from Human Feedback (RLHF) in settings where multiple labelers may strategically misreport feedback to steer the learned policy toward their own preferences. We show that existing RLHF algorithms, including recent pluralistic methods, are not strategyproof, and that even a single strategic labeler can cause arbitrarily large misalignment with social welfare. Moreover, we prove that, in the worst case, any strategyproof RLHF algorithm must perform k-times worse than the optimal policy, where k is the number of labelers. This suggests a fundamental trade-off between incentive alignment (ensuring labelers report truthfully) and policy alignment (maximizing social welfare). To address this, we propose the Pessimistic Median of MLEs algorithm, which, under appropriate policy coverage assumptions, is approximately strategyproof and converges to the optimal policy as the number of labelers and samples increases. Our results apply to both contextual bandits and Markov decision processes.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 39 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

30 of 39 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuning22 Dec
  2. 2024Robust Reinforcement Learning from Corrupted Human Feedback21 Jun
  3. 2024Axioms for AI Alignment from Human Feedback23 May
  4. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation30 Apr
  5. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
  6. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  7. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  8. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  9. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  11. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2023Whose Opinions Do Language Models Reflect?cited by 773
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  3. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  4. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  5. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  6. 2024A Roadmap to Pluralistic Alignmentcited by 184
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  8. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  9. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  10. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  11. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  12. 2024Exploiting LLM Quantizationcited by 65
6 more
  1. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 54
  2. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  3. 2024Truthful Aggregation of LLMs with an Application to Online Advertisingcited by 29
  4. 2024Mechanism Design for LLM Fine-tuning with Multiple Reward Modelscited by 18
  5. 2024Policy Aggregationcited by 8
  6. 2024Strategic Linear Contextual Banditscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 4
  2. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  3. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 1
  4. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  5. 2026Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2025Reinforcement Learning from Human Feedback
  4. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory
  5. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game
  6. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
  7. 2025Maximizing the efficiency of human feedback in AI alignment: a comparative analysis
  8. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling