mlsanityData to 7 Oct 2026

Batch Active Learning of Reward Functions from Human Preferences

arXiv 2402.15757 ↗First posted 24 Feb 2024cs.LG cs.AI cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

Data generation and labeling are often expensive in robot learning. Preference-based learning is a concept that enables reliable labeling by querying users with preference questions. Active querying methods are commonly employed in preference-based learning to generate more informative data at the expense of parallelization and computation time. In this paper, we develop a set of novel algorithms, batch active preference-based learning methods, that enable efficient learning of reward functions using as few data samples as possible while still having short query generation times and also retaining parallelizability. We introduce a method based on determinantal point processes (DPP) for active batch generation and several heuristic-based alternatives. Finally, we present our experimental results for a variety of robotics tasks in simulation. Our results suggest that our batch active learning algorithm requires only a few queries that are computed in a short amount of time. We showcase one of our algorithms in a study to learn human users' preferences.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 91 references that are in the map. First posted on arXiv 24 Feb 2024.

Builds on

37 of 91 references.

References · 37

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  5. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  6. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  8. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  9. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  10. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  11. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  12. 2019Batch Active Learning Using Determinantal Point Processescited by 42
25 more
  1. 2017Active Learning for Convolutional Neural Networks: A Core-Set Approachcited by 34
  2. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  3. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  4. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  5. 2016Monte Carlo Markov Chain Algorithms for Sampling Strongly Rayleigh Distributions and Determinantal Point Processescited by 24
  6. 2021Learning Reward Functions from Scale Feedbackcited by 24
  7. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 23
  8. 2020Active Preference Learning using Maximum Regretcited by 21
  9. 2020Active Reinforcement Learning: Observing Rewards at a Costcited by 20
  10. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
  11. 2018Active Mini-Batch Sampling using Repulsive Point Processescited by 19
  12. 2020ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapescited by 19
  13. 2014Dissimilarity-based Sparse Subset Selectioncited by 18
  14. 2016A First Look at Ad-block Detection: A New Arms Race on the Webcited by 15
  15. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  16. 2023Active Reward Learning from Online Preferencescited by 12
  17. 2014Randomized Rounding for the Largest Simplex Problemcited by 10
  18. 2017Medoids in almost linear time via multi-armed banditscited by 10
  19. 2018Single Shot Active Learning using Pseudo Annotatorscited by 9
  20. 2020Interactive Robot Training for Non-Markov Taskscited by 9
  21. 2021Preference-based Learning of Reward Function Featurescited by 9
  22. 2016Fast Mixing Markov Chains for Strongly Rayleigh Measures, DPPs, and Constrained Samplingcited by 8
  23. 2018Log-Concave Polynomials II: High-Dimensional Walks and an FPRAS for Counting Bases of a Matroidcited by 7
  24. 2016Local Search for Max-Sum Diversificationcited by 3
  25. 2023Preference Elicitation with Soft Attributes in Interactive Recommendationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 2.

8 papers.

  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2
  3. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
  4. 2025Optimal Interactive Learning on the Job via Facility Location Planningcited by 2
  5. 2026MetaRanker: Human-in-the-loop Active Ranking for Metalens Image Qualitycited by 1
  6. 2024Learning Quantitative Automata Modulo Theoriescited by 0
  7. 2025Learning Acrobatic Flight from Preferencescited by 0
  8. 2026Specifying Reward Functions for RL Without Environment Samplingcited by 0