mlsanityData to 14 Oct 2026

Beyond Reward: Offline Preference-guided Policy Optimization

arXiv 2305.16217 ↗First posted 25 May 2023cs.LGcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

This study focuses on the topic of offline preference-based reinforcement learning (PbRL), a variant of conventional reinforcement learning that dispenses with the need for online interaction or specification of reward functions. Instead, the agent is provided with fixed offline trajectories and human preferences between pairs of trajectories to extract the dynamics and task information, respectively. Since the dynamics and task information are orthogonal, a naive approach would involve using preference-based reward learning followed by an off-the-shelf offline RL algorithm. However, this requires the separate learning of a scalar reward function, which is assumed to be an information bottleneck of the learning process. To address this issue, we propose the offline preference-guided policy optimization (OPPO) paradigm, which models offline trajectories and preferences in a one-step process, eliminating the need for separately learning a reward function. OPPO achieves this by introducing an offline hindsight information matching objective for optimizing a contextual policy and a preference modeling objective for finding the optimal context. OPPO further integrates a well-performing decision policy by optimizing the two objectives iteratively. Our empirical results demonstrate that OPPO effectively models offline preferences and outperforms prior competing baselines, including offline RL algorithms performed over either true or pseudo reward function specifications. Our code is available on the project website: https://sites.google.com/view/oppo-icml-2023 .

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 69 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

46 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Models of human preference for learning reward functions5 Jun
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  5. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 40

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
28 more
  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  3. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  4. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  5. 2017Inverse Reward Designcited by 249
  6. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  7. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  8. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  9. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  10. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  11. 2013Risk-sensitive Reinforcement Learningcited by 130
  12. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  13. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  14. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  15. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  16. 2019Reward-Conditioned Policiescited by 102
  17. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  18. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  19. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  20. 2023Behavior Proximal Policy Optimizationcited by 43
  21. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  22. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  23. 2020Planning from Pixels using Inverse Dynamics Modelscited by 36
  24. 2019Learning to Reach Goals via Iterated Supervised Learningcited by 26
  25. 2021Visualizing Data using GTSNEcited by 24
  26. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 17
  27. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  28. 2021Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 13; 2025: 11; 2026: 3.

35 papers.

Sort:

By area:

  1. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  2. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  3. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  4. 2024Improving Attributed Text Generation of Large Language Models via Preference Learningcited by 23
  5. 2024Optimal Design for Human Preference Elicitationcited by 17
  6. 2025Proximal Supervised Fine-Tuningcited by 16
  7. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  8. 2023CEIL: Generalized Contextual Imitation Learningcited by 13
  9. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  10. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  11. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  12. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
23 more
  1. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  2. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  3. 2024Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfecited by 5
  4. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  5. 2023Conditions on Preference Relations that Guarantee the Existence of Optimal Policiescited by 3
  6. 2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3
  7. 2024Hindsight Preference Learning for Offline Preference-based Reinforcement Learningcited by 3
  8. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  9. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  10. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  11. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  12. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  13. 2023Safe and Interpretable Estimation of Optimal Treatment Regimescited by 1
  14. 2025Behavior Preference Regression for Offline Reinforcement Learningcited by 1
  15. 2025Offline Safe Policy Optimization From Heterogeneous Feedbackcited by 1
  16. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 1
  17. 2024Preference-Guided Reinforcement Learning for Efficient Explorationcited by 0
  18. 2024UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrationscited by 0
  19. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0
  20. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
  21. 2025TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learningcited by 0
  22. 2026Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitationcited by 0
  23. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Online Policy Learning from Offline Preferences
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  3. 2024Preference Elicitation for Offline Reinforcement Learning
  4. 2023Direct Preference-based Policy Optimization without Reward Modeling
  5. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  6. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  7. 2024Advances in Preference-based Reinforcement Learning: A Review
  8. 2026SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling