mlsanityData to 14 Oct 2026

Learning Guarantee of Reward Modeling Using Deep Neural Networks

arXiv 2505.06601 ↗First posted 10 May 2025stat.ML cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this work, we study the learning theory of reward modeling with pairwise comparison data using deep neural networks. We establish a novel non-asymptotic regret bound for deep reward estimators in a non-parametric setting, which depends explicitly on the network architecture. Furthermore, to underscore the critical importance of clear human beliefs, we introduce a margin-type condition that assumes the conditional winning probability of the optimal action in pairwise comparisons is significantly distanced from 1/2. This condition enables a sharper regret bound, which substantiates the empirical efficiency of Reinforcement Learning from Human Feedback and highlights clear human beliefs in its success. Notably, this improvement stems from high-quality pairwise comparison data implied by the margin-type condition, is independent of the specific estimators used, and thus applies to various learning algorithms and models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 45 references that are in the map. First posted on arXiv 10 May 2025.

Builds on

29 of 45 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd19 Nov
  2. 2024Dual Active Learning for Reinforcement Learning from Human Feedback3 Oct
  3. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback23 Jul
  4. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  6. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  7. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model3 Oct
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  5. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  6. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  7. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  8. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  9. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  10. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 61
  11. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  12. 2018Fast convergence rates of deep neural networks for classificationcited by 46
7 more
  1. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  2. 2020Efficient active learning of sparse halfspaces with arbitrary bounded noisecited by 29
  3. 2021Learning General Halfspaces with General Massart Noise under the Gaussian Distributioncited by 27
  4. 2022Generalization Error Bounds for Deep Neural Networks Trained by SGDcited by 14
  5. 2024On Extending Direct Preference Optimization to Accommodate Tiescited by 8
  6. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  7. 2025Approximation Bounds for Transformer Networks with Application to Regressioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  2. 2025Reward Models in Deep Reinforcement Learning: A Survey
  3. 2025Capturing Individual Human Preferences with Reward Features
  4. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2025Debiasing Reward Models by Representation Learning with Guarantees
  7. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  8. 2018Reward learning from human preferences and demonstrations in Atari