mlsanityData to 7 Oct 2026

Reinforcement Learning from Human Feedback: A Statistical Perspective

arXiv 2604.02507 ↗First posted 2 Apr 2026stat.ML cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as a central framework for aligning large language models (LLMs) with human preferences. Despite its practical success, RLHF raises fundamental statistical questions because it relies on noisy, subjective, and often heterogeneous feedback to learn reward models and optimize policies. This survey provides a statistical perspective on RLHF, focusing primarily on the LLM alignment setting. We introduce the main components of RLHF, including supervised fine-tuning, reward modeling, and policy optimization, and relate them to familiar statistical ideas such as Bradley-Terry-Luce (BTL) model, latent utility estimation, active learning, experimental design, and uncertainty quantification. We review methods for learning reward functions from pairwise preference data and for optimizing policies through both two-stage RLHF pipelines and emerging one-stage approaches such as direct preference optimization. We further discuss recent extensions including reinforcement learning from AI feedback, inference-time algorithms, and reinforcement learning from verifiable rewards, as well as benchmark datasets, evaluation protocols, and open-source frameworks that support RLHF research. We conclude by highlighting open challenges in RLHF. An accompanying GitHub demo https://github.com/Pangpang-Liu/RLHF_demo illustrates key components of the RLHF pipeline.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 85 of its 95 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

85 of 95 references.

Closest ancestors · 30

Same direction, newest first

  1. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback2 Dec
  2. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspective29 May
  3. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO26 May
  4. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedback27 Apr
  5. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning3 Apr
  6. 2024Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback27 Dec
  7. 2024Optimal Design for Reward Modeling in RLHF22 Oct
  8. 2024How to Evaluate Reward Models for RLHF18 Oct
  9. 2024Dual Active Learning for Reinforcement Learning from Human Feedback3 Oct
  10. 2024Scalable Ensembling For Mitigating Reward Overoptimisation3 Jun
  11. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation30 Apr
  12. 2024More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness29 Apr
18 more
  1. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
  2. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  3. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  4. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  5. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  6. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  9. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2023Active Reward Learning from Multiple Teachers2 Mar
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  13. 2022The Expertise Problem: Learning from Specialized Feedback12 Nov
  14. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  15. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  16. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  17. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  18. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2024A Survey on LLM-as-a-Judgecited by 1,384
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
38 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2025Group Sequence Policy Optimizationcited by 788
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  6. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  7. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  8. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  9. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  10. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  11. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  12. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  13. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  14. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 52
  15. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  16. 2021Uncertainty quantification in the Bradley-Terry-Luce modelcited by 28
  17. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  18. 2022Ranking Inferences Based on the Top Choice of Multiway Comparisonscited by 22
  19. 2025PairJudge RM: Perform Best-of-N Sampling with Knockout Tournamentcited by 21
  20. 2021Lagrangian Inference for Ranking Problemscited by 19
  21. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  22. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  23. 2024Optimal Design for Human Preference Elicitationcited by 17
  24. 2022Uncertainty Quantification of MLE for Entity Ranking with Covariatescited by 16
  25. 2025Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysiscited by 15
  26. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  27. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  28. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMscited by 10
  29. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  30. 2026Audit Trails for Accountability in Large Language Modelscited by 10
  31. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  32. 2023Active teacher selection for reward learningcited by 7
  33. 2025RLTHF: Targeted Human Feedback for LLM Alignmentcited by 7
  34. 2024Covariate Assisted Entity Ranking with Sparse Intrinsic Scorescited by 6
  35. 2024Confidence Diagram of Nonparametric Ranking for Uncertainty Assessment in Large Language Models Evaluationcited by 5
  36. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraintscited by 4
  37. 2026A Statistical Framework for Alignment with Biased AI Feedbackcited by 4
  38. 2025Efficient Inference for Covariate-adjusted Bradley-Terry Model with Covariate Shiftcited by 3

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  2. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  3. 2026Variance-aware Reward Modeling with Anchor Guidancecited by 0
  4. 2026Mathematical methods of reinforcement learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  2. 2025Reinforcement Learning from Human Feedback
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  6. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  7. 2023The History and Risks of Reinforcement Learning and Human Feedback
  8. 2023A Survey of Reinforcement Learning from Human Feedback