mlsanityData to 14 Oct 2026

Query-Policy Misalignment in Preference-Based Reinforcement Learning

arXiv 2305.17400 ↗First posted 27 May 2023cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) provides a natural way to align RL agents' behavior with human desired outcomes, but is often restrained by costly human feedback. To improve feedback efficiency, most existing PbRL methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitively, we find that this may not necessarily lead to improved performance. To unravel this mystery, we identify a long-neglected issue in the query selection schemes of existing PbRL studies: Query-Policy Misalignment. We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents' interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. We show that this issue can be effectively addressed via near on-policy query and a specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. Simple yet elegant, our method can be easily incorporated into existing approaches by changing only a few lines of code. We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

33 of 55 references.

References · 33

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2018DeepMind Control Suitecited by 956
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  8. 2020Reinforcement Learning with Augmented Datacited by 468
  9. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  10. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  11. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  12. 2017Inverse Reward Designcited by 249
21 more
  1. 2016Value Iteration Networkscited by 160
  2. 2017Value Prediction Networkcited by 117
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2017A Deeper Look at Experience Replaycited by 105
  5. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  6. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 96
  7. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 94
  8. 2016The Predictron: End-To-End Learning and Planningcited by 93
  9. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  10. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  11. 2021On the Expressivity of Markov Rewardcited by 71
  12. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  13. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
  14. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 53
  15. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  16. 2015Learning Contact-Rich Manipulation Skills with Guided Policy Searchcited by 38
  17. 2015One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priorscited by 26
  18. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  19. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  20. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  21. 2023TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matchingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  2. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 20
  3. 2024Instruction-Guided Visual Maskingcited by 17
  4. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  5. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  6. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  7. 2025DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisitioncited by 1
  8. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learningcited by 1
  9. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Roboticscited by 1
  10. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 1
  11. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0
  12. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learningcited by 0
3 more
  1. 2026Reward Learning through Ranking Mean Squared Errorcited by 0
  2. 2026The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learningcited by 0
  3. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
  2. 2023Direct Preference-based Policy Optimization without Reward Modeling
  3. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  4. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  7. 2024Advances in Preference-based Reinforcement Learning: A Review
  8. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration