mlsanityData to 7 Oct 2026

OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration

arXiv 2604.02349 ↗First posted 19 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) can help avoid sophisticated reward designs and align better with human intentions, showing great promise in various real-world applications. However, obtaining human feedback for preferences can be expensive and time-consuming, which forms a strong barrier for PbRL. In this work, we address the problem of low query efficiency in offline PbRL, pinpointing two primary reasons: inefficient exploration and overoptimization of learned reward functions. In response to these challenges, we propose a novel algorithm, Offline PbRL via In-Dataset Exploration (OPRIDE), designed to enhance the query efficiency of offline PbRL. OPRIDE consists of two key features: a principled exploration strategy that maximizes the informativeness of the queries and a discount scheduling mechanism aimed at mitigating overoptimization of the learned reward functions. Through empirical evaluations, we demonstrate that OPRIDE significantly outperforms prior methods, achieving strong performance with notably fewer queries. Moreover, we provide theoretical guarantees of the algorithm's efficiency. Experimental results across various locomotion, manipulation, and navigation tasks underscore the efficacy and versatility of our approach.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

43 of 57 references.

References · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  10. 2018Is Q-learning Provably Efficient?cited by 485
  11. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
31 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  3. 2013Learning to Optimize Via Posterior Samplingcited by 253
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  5. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  6. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  7. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  8. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  9. 2023Is RLHF More Difficult than Standard RL?cited by 96
  10. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  11. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  12. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  13. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  14. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  15. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  16. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  17. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  18. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  19. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  20. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  21. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  22. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  23. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  24. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  25. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  26. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
  27. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  28. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  29. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  30. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  31. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4

Built on by

Not cited yet.