mlsanityData to 14 Oct 2026

Optimal Design for Reward Modeling in RLHF

arXiv 2410.17055 ↗First posted 22 Oct 2024cs.LG stat.MLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has become a popular approach to align language models (LMs) with human preferences. This method involves collecting a large dataset of human pairwise preferences across various text generations and using it to infer (implicitly or explicitly) a reward model. Numerous methods have been proposed to learn the reward model and align a LM with it. However, the costly process of collecting human preferences has received little attention and could benefit from theoretical insights. This paper addresses this issue and aims to formalize the reward training model in RLHF. We frame the selection of an effective dataset as a simple regret minimization task, using a linear contextual dueling bandit method. Given the potentially large number of arms, this approach is more coherent than the best-arm identification setting. We then propose an offline framework for solving this problem. Under appropriate assumptions - linearity of the reward model in the embedding space, and boundedness of the reward parameter - we derive bounds on the simple regret. Finally, we provide a lower bound that matches our upper bound up to constant and logarithmic terms. To our knowledge, this is the first theoretical contribution in this area to provide an offline approach as well as worst-case guarantees.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 66 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

51 of 66 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison15 Sep
  2. 2024Off-Policy Evaluation from Logged Human Feedback14 Jun
  3. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  4. 2024Dataset Reset Policy Optimization for RLHF12 Apr
  5. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  6. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  7. 2023Nash Learning from Human Feedback1 Dec
  8. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback8 Aug
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  10. 2023Is RLHF More Difficult than Standard RL?25 Jun
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
2 more
  1. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
25 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  3. 2020Improved Optimistic Algorithms for Logistic Banditscited by 118
  4. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  5. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  6. 2015Contextual Dueling Banditscited by 101
  7. 2014Best-Arm Identification in Linear Banditscited by 100
  8. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  9. 2020Gamification of Pure Exploration for Linear Banditscited by 63
  10. 2020Instance-Wise Minimax-Optimal Algorithms for Logistic Banditscited by 57
  11. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  12. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 52
  13. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 37
  14. 2016Multidimensional Binary Search for Contextual Decision-Makingcited by 27
  15. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 24
  16. 2023Improved Regret Bounds of (Multinomial) Logistic Bandits via Regret-to-Confidence-Set Conversioncited by 24
  17. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  18. 2023Variance-Aware Regret Bounds for Stochastic Contextual Dueling Banditscited by 22
  19. 2019Old Dog Learns New Tricks: Randomized UCB for Bandit Problemscited by 21
  20. 2024Nearly Minimax Optimal Regret for Multinomial Logistic Banditcited by 20
  21. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  22. 2024Optimal Design for Human Preference Elicitationcited by 17
  23. 2022Human Preferences as Dueling Banditscited by 14
  24. 2021Teaching an Active Learner with Contrastive Examplescited by 12
  25. 2024OPTune: Efficient Online Preference Tuningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 7.

25 papers.

Sort:

By area:

  1. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  2. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  3. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  4. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 15
  5. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 12
  6. 2025Active Learning for Direct Preference Optimizationcited by 10
  7. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedbackcited by 9
  8. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  9. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 5
  10. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  11. 2025FraPPE: Fast and Efficient Preference-based Pure Explorationcited by 4
  12. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
13 more
  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  2. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
  3. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
  4. 2025The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneitycited by 2
  5. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  6. 2025Reasoning without Regretcited by 0
  7. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  8. 2026Microeconomic Foundations of Multi-Agent Learningcited by 0
  9. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignmentcited by 0
  10. 2026Adaptive Simulation Experiment for LLM Policy Optimizationcited by 0
  11. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuningcited by 0
  12. 2026How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysiscited by 0
  13. 2026Which Pairs to Compare for LLM Post-Training?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  3. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2025Greedy Sampling Is Provably Efficient for RLHF
  7. 2024How to Evaluate Reward Models for RLHF
  8. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning