mlsanityData to 7 Oct 2026

Preference-based Learning of Reward Function Features

arXiv 2103.02727 ↗First posted 3 Mar 2021cs.RO cs.HCcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based learning of reward functions, where the reward function is learned using comparison data, has been well studied for complex robotic tasks such as autonomous driving. Existing algorithms have focused on learning reward functions that are linear in a set of trajectory features. The features are typically hand-coded, and preference-based learning is used to determine a particular user's relative weighting for each feature. Designing a representative set of features to encode reward is challenging and can result in inaccurate models that fail to model the users' preferences or perform the task properly. In this paper, we present a method to learn both the relative weighting among features as well as additional features that help encode a user's reward function. The additional features are modeled as a neural network that is trained on the data from pairwise comparison queries. We apply our methods to a driving scenario used in previous work and compare the predictive power of our method to that of only hand-coded features. We perform additional analysis to interpret the learned features and examine the optimal trajectories. Our results show that adding an additional learned feature to the reward model enhances both its predictive power and expressiveness, producing unique results for each user.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 24 references that are in the map. First posted on arXiv 3 Mar 2021.

Builds on

12 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  2. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 9

  1. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  2. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  3. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  4. 2019Preferences Implicit in the State of the Worldcited by 33
  5. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  6. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 23
  7. 2020Active Preference Learning using Maximum Regretcited by 21
  8. 2019Learning an Urban Air Mobility Encounter Model from Expert Preferencescited by 12
  9. 2019Bayesian Active Learning for Collaborative Task Specification Using Equivalence Regionscited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 2.

9 papers.

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  2. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  3. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  4. 2023Active Reward Learning from Online Preferencescited by 12
  5. 2023Human-oriented Representation Learning for Robotic Manipulationcited by 10
  6. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  7. 2022Receding Horizon Planning with Rule Hierarchies for Autonomous Vehiclescited by 7
  8. 2024Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignmentcited by 6
  9. 2022Learning Preferences for Interactive Autonomycited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  2. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  3. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  4. 2023Learning Optimal Advantage from Preferences and Mistaking it for Reward
  5. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning
  6. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  7. 2023On The Fragility of Learned Reward Functions
  8. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferences