mlsanityData to 7 Oct 2026

Data Driven Reward Initialization for Preference based Reinforcement Learning

arXiv 2302.08733 ↗First posted 17 Feb 2023cs.LG cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based Reinforcement Learning (PbRL) methods utilize binary feedback from the human in the loop (HiL) over queried trajectory pairs to learn a reward model in an attempt to approximate the human's underlying reward function capturing their preferences. In this work, we investigate the issue of a high degree of variability in the initialized reward models which are sensitive to random seeds of the experiment. This further compounds the issue of degenerate reward functions PbRL methods already suffer from. We propose a data-driven reward initialization method that does not add any additional cost to the human in the loop and negligible cost to the PbRL agent and show that doing so ensures that the predicted rewards of the initialized reward model are uniform in the state space and this reduces the variability in the performance of the method across multiple runs and is shown to improve the overall performance compared to other initialization methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 35 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

18 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences17 Oct
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 14

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  5. 2017Inverse Reward Designcited by 246
  6. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  7. 2017On weight initialization in deep neural networkscited by 67
  8. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  9. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  10. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  11. 2022Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loopcited by 5
  12. 2022Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansioncited by 4
2 more
  1. 2019Making Smart Homes Smarter: Optimizing Energy Consumption with Human in the Loopcited by 3
  2. 2021Computing Policies That Account For The Effects Of Human Agent Uncertainty During Execution In Markov Decision Processescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.

3 papers.

  1. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learningcited by 1
  2. 2023Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environmentscited by 0
  3. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  2. 2023Direct Preference-based Policy Optimization without Reward Modeling
  3. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  4. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  5. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  6. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  8. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning