mlsanityData to 7 Oct 2026

Active Preference-Based Gaussian Process Regression for Reward Learning

arXiv 2005.02575 ↗First posted 6 May 2020cs.RO cs.AI cs.LGcited by 70

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Designing reward functions is a challenging problem in AI and robotics. Humans usually have a difficult time directly specifying all the desirable behaviors that a robot needs to optimize. One common approach is to learn reward functions from collected expert demonstrations. However, learning reward functions from demonstrations introduces many challenges: some methods require highly structured models, e.g. reward functions that are linear in some predefined set of features, while others adopt less structured reward functions that on the other hand require tremendous amount of data. In addition, humans tend to have a difficult time providing demonstrations on robots with high degrees of freedom, or even quantifying reward values for given demonstrations. To address these challenges, we present a preference-based learning approach, where as an alternative, the human feedback is only in the form of comparisons between trajectories. Furthermore, we do not assume highly constrained structures on the reward function. Instead, we model the reward function using a Gaussian Process (GP) and propose a mathematical formulation to actively find a GP using only human preferences. Our approach enables us to tackle both inflexibility and data-inefficiency problems within a preference-based learning framework. Our results in simulations and a user study suggest that our approach can efficiently learn expressive reward functions for robotics tasks.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 76 references that are in the map. First posted on arXiv 6 May 2020.

Builds on

24 of 76 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2016OpenAI Gymcited by 1,684
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2013Gaussian Processes for Big Datacited by 377
  4. 2018When Gaussian Process Meets Big Data: A Review of Scalable GPscited by 205
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  6. 2017Third-Person Imitation Learningcited by 103
  7. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
  8. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  9. 2017Preferential Bayesian Optimizationcited by 72
  10. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  11. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  12. 2015Shared Autonomy via Hindsight Optimizationcited by 46
10 more
  1. 2019Batch Active Learning Using Determinantal Point Processescited by 42
  2. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  3. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  4. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 23
  5. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  6. 2016A First Look at Ad-block Detection: A New Arms Race on the Webcited by 15
  7. 2019The Green Choice: Learning and Influencing Human Decisions on Shared Roadscited by 14
  8. 2019Learning an Urban Air Mobility Encounter Model from Expert Preferencescited by 12
  9. 2020Human Preference-Based Learning for High-dimensional Optimization of Exoskeleton Walking Gaitscited by 6
  10. 2019Bayesian Active Learning for Collaborative Task Specification Using Equivalence Regionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 11; 2022: 5; 2023: 14; 2024: 20; 2025: 12; 2026: 7.

70 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  3. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  4. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  5. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  6. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  7. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  8. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  9. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  10. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  11. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  12. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
38 more
  1. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  2. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  3. 2021Learning Multimodal Rewards from Rankingscited by 41
  4. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  5. 2021What are you optimizing for? Aligning Recommender Systems with Human Valuescited by 28
  6. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  7. 2021Learning Reward Functions from Scale Feedbackcited by 24
  8. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  9. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
  10. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 19
  11. 2024Optimal Design for Human Preference Elicitationcited by 17
  12. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  13. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  14. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  15. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  16. 2023Active Reward Learning from Online Preferencescited by 12
  17. 2024The Role of Predictive Uncertainty and Diversity in Embodied AI and Robot Learningcited by 11
  18. 2021Preference-based Learning of Reward Function Featurescited by 9
  19. 2021Preference learning along multiple criteria: A game-theoretic perspectivecited by 9
  20. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  21. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  22. 2024Reinforcement Learning from Human Feedback: Whose Culture, Whose Values, Whose Perspectives?cited by 8
  23. 2024DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignmentcited by 7
  24. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  25. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  26. 2024Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimationcited by 6
  27. 2025Optimistic Task Inference for Behavior Foundation Modelscited by 5
  28. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
  29. 2021Incentivizing Efficient Equilibria in Traffic Networks with Mixed Autonomycited by 3
  30. 2022Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learningcited by 3
  31. 2023Optimizing Algorithms From Pairwise User Preferencescited by 3
  32. 2023Deep Reinforcement Learning from Hierarchical Preference Designcited by 3
  33. 2023Preference Elicitation with Soft Attributes in Interactive Recommendationcited by 3
  34. 2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3
  35. 2025Learning Implicit Social Navigation Behavior using Deep Inverse Reinforcement Learningcited by 3
  36. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  37. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 3
  38. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning Preferences for Interactive Autonomy
  2. 2026Freeform Preference Learning for Robotic Manipulation
  3. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  4. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  5. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regression
  6. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction
  7. 2020Gaussian-Process-based Robot Learning from Demonstration
  8. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback