mlsanityData to 7 Oct 2026

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

arXiv 2205.12401 ↗First posted 24 May 2022cs.LG cs.AIcited by 60

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL methods are able to learn a more flexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher's preferences between two clips of behaviors. However, poor feedback-efficiency still remains a problem in current preference-based RL algorithms, as tailored human feedback is very expensive. To handle this issue, previous methods have mainly focused on improving query selection and policy initialization. At the same time, recent exploration methods have proven to be a recipe for improving sample-efficiency in RL. We present an exploration method specifically for preference-based RL algorithms. Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward. Specifically, we utilize disagreement across ensemble of learned reward models. Our intuition is that disagreement in learned reward model reflects uncertainty in tailored human feedback and could be useful for exploration. Our experiments show that exploration bonus from uncertainty in learned reward improves both feedback- and sample-efficiency of preference-based RL algorithms on complex robot manipulation tasks from MetaWorld benchmarks, compared with other existing exploration methods that measure the novelty of state visitation.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 47 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

33 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  2. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  4. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
16 more
  1. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  2. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  3. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  4. 2019Self-Supervised Exploration via Disagreementcited by 263
  5. 2017Count-Based Exploration with Neural Density Modelscited by 255
  6. 2017Inverse Reward Designcited by 246
  7. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  8. 2019Efficient Exploration via State Marginal Matchingcited by 186
  9. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  10. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  11. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  12. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  13. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  14. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  15. 2020Quantifying Differences in Reward Functionscited by 62
  16. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 15; 2024: 20; 2025: 13; 2026: 8.

60 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  3. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  4. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  5. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
  6. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  7. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  8. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  9. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  10. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  11. 2023Proportional Aggregation of Preferences for Sequential Decision Makingcited by 17
  12. 2024RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviorscited by 16
38 more
  1. 2022Transformers are Adaptable Task Plannerscited by 15
  2. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  3. 2023Rating-based Reinforcement Learningcited by 14
  4. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  5. 2025Efficient Process Reward Model Training via Active Learningcited by 14
  6. 2023Active Reward Learning from Multiple Teacherscited by 13
  7. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  8. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
  9. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  10. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
  11. 2024Towards Reliable Alignment: Uncertainty-aware RLHFcited by 7
  12. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  13. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  14. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  15. 2024Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimationcited by 6
  16. 2024Safety through feedback in Constrained RLcited by 5
  17. 2022Skill-Based Reinforcement Learning with Intrinsic Reward Matchingcited by 4
  18. 2022Efficient Meta Reinforcement Learning for Preference-based Fast Adaptationcited by 4
  19. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
  20. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functionscited by 3
  21. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  22. 2025Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedbackcited by 3
  23. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  24. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  25. 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
  26. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
  27. 2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2
  28. 2023Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scorescited by 1
  29. 2023STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularizationcited by 1
  30. 2023Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testingcited by 1
  31. 2023Resilient Constrained Reinforcement Learningcited by 1
  32. 2024Learning Hidden Subgoals under Temporal Ordering Constraints in Reinforcement Learningcited by 1
  33. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Roboticscited by 1
  34. 2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learningcited by 1
  35. 2023Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networkscited by 0
  36. 2024Preference-Guided Reinforcement Learning for Efficient Explorationcited by 0
  37. 2024Dual Action Policy for Robust Sim-to-Real Reinforcement Learningcited by 0
  38. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  2. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  5. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning
  6. 2023Is RLHF More Difficult than Standard RL?
  7. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  8. 2023Contrastive Preference Learning: Learning from Human Feedback without RL