mlsanityData to 7 Oct 2026

A Generalized Acquisition Function for Preference-based Reward Learning

arXiv 2403.06003 ↗First posted 9 Mar 2024cs.RO cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Preference-based reward learning is a popular technique for teaching robots and autonomous systems how a human user wants them to perform a task. Previous works have shown that actively synthesizing preference queries to maximize information gain about the reward function parameters improves data efficiency. The information gain criterion focuses on precisely identifying all parameters of the reward function. This can potentially be wasteful as many parameters may result in the same reward, and many rewards may result in the same behavior in the downstream tasks. Instead, we show that it is possible to optimize for learning the reward function up to a behavioral equivalence class, such as inducing the same ranking over behaviors, distribution over choices, or other related definitions of what makes two rewards similar. We introduce a tractable framework that can capture such definitions of similarity. Our experiments in a synthetic environment, an assistive robotics environment with domain transfer, and a natural language processing problem with real datasets demonstrate the superior performance of our querying method over the state-of-the-art information gain method.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 9 Mar 2024.

Builds on

22 of 40 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  7. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  8. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  9. 2016The Off-Switch Gamecited by 97
  10. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  11. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  12. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
10 more
  1. 2020Quantifying Differences in Reward Functionscited by 62
  2. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  3. 2019Assistive Gym: A Physics Simulation Framework for Assistive Roboticscited by 44
  4. 2021Learning Multimodal Rewards from Rankingscited by 41
  5. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  6. 2021Learning Reward Functions from Scale Feedbackcited by 24
  7. 2020Active Preference Learning using Maximum Regretcited by 21
  8. 2020ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapescited by 19
  9. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
  10. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimizationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0