mlsanityData to 7 Oct 2026

Contextual Online Uncertainty-Aware Preference Learning for Human Feedback

arXiv 2504.19342 ↗First posted 27 Apr 2025stat.ML cs.LG stat.MEcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm in artificial intelligence to align large models with human preferences. In this paper, we propose a novel statistical framework to simultaneously conduct the online decision-making and statistical inference on the optimal model using human preference data based on dynamic contextual information. Our approach introduces an efficient decision strategy that achieves both the optimal regret bound and the asymptotic distribution of the estimators. A key challenge in RLHF is handling the dependent online human preference outcomes with dynamic contexts. To address this, in the methodological aspect, we propose a two-stage algorithm starting with ε-greedy followed by exploitations; in the theoretical aspect, we tailor anti-concentration inequalities and matrix martingale concentration techniques to derive the uniform estimation rate and asymptotic normality of the estimators using dependent samples from both stages. Extensive simulation results demonstrate that our method outperforms state-of-the-art strategies. We apply the proposed framework to analyze the human preference data for ranking large language models on the Massive Multitask Language Understanding dataset, yielding insightful results on the performance of different large language models for medical anatomy knowledge.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 79 references that are in the map. First posted on arXiv 27 Apr 2025.

Builds on

50 of 79 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
  2. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  4. 2023Demonstration-Regularized RL26 Oct
  5. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  6. 2023Is RLHF More Difficult than Standard RL?25 Jun
  7. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023Reward Collapse in Aligning Large Language Models28 May
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2014Convex Optimization: Algorithms and Complexitycited by 569
  8. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  9. 2019Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Searchcited by 137
  10. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  11. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  12. 2018Practical Contextual Bandits with Regression Oraclescited by 92
26 more
  1. 2017Mostly Exploration-Free Algorithms for Contextual Banditscited by 85
  2. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  3. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  4. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  5. 2020Sequential Batch Learning in Finite-Action Linear Contextual Banditscited by 63
  6. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 57
  7. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  8. 2019Nearly Minimax-Optimal Regret for Linearly Parameterized Banditscited by 54
  9. 2017MNL-Bandit: A Dynamic Learning Approach to Assortment Selectioncited by 50
  10. 2018Adversarial Personalized Ranking for Recommendationcited by 48
  11. 2017Spectral Method and Regularized MLE Are Both Optimal for Top-K Rankingcited by 46
  12. 2020Fast Rates for Contextual Linear Optimizationcited by 43
  13. 2017Accurate Inference for Adaptive Linear Modelscited by 39
  14. 2018Dynamic Assortment Optimization with Changing Contextual Informationcited by 28
  15. 2021Uncertainty quantification in the Bradley-Terry-Luce modelcited by 28
  16. 2019Smooth Contextual Bandits: Bridging the Parametric and Non-differentiable Regret Regimescited by 25
  17. 2017Non-Stationary Bandits with Habituation and Recovery Dynamicscited by 23
  18. 2022Ranking Inferences Based on the Top Choice of Multiway Comparisonscited by 22
  19. 2021Near-optimal inference in adaptive linear regressioncited by 20
  20. 2021Lagrangian Inference for Ranking Problemscited by 19
  21. 2020Statistical Inference for Online Decision-Making: In a Contextual Bandit Settingcited by 17
  22. 2022Uncertainty Quantification of MLE for Entity Ranking with Covariatescited by 16
  23. 2023Efficient Explorative Key-term Selection Strategies for Conversational Contextual Banditscited by 12
  24. 2024Studying LLM Performance on Closed- and Open-source Datacited by 11
  25. 2023Spectral Ranking Inferences based on General Multiway Comparisonscited by 10
  26. 2022A Simple and Optimal Policy Design with Safety against Heavy-Tailed Risk for Stochastic Banditscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  2. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedbackcited by 7
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  4. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantificationcited by 3
  5. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  6. 2026Perturbation is All You Need for Extrapolating Language Modelscited by 1
  7. 2025Preference-Based Dynamic Ranking Structure Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  2. 2024Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback
  3. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  4. 2025Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference
  5. 2026Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
  6. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  7. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  8. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?