mlsanityData to 7 Oct 2026

Offline Preference-Based Apprenticeship Learning

arXiv 2107.09251 ↗First posted 20 Jul 2021cs.LGcited by 17

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning a reward function from human preferences is challenging as it typically requires having a high-fidelity simulator or using expensive and potentially unsafe actual physical rollouts in the environment. However, in many tasks the agent might have access to offline data from related tasks in the same target environment. While offline data is increasingly being used to aid policy optimization via offline RL, our observation is that it can be a surprisingly rich source of information for preference learning as well. We propose an approach that uses an offline dataset to craft preference queries via pool-based active learning, learns a distribution over reward functions, and optimizes a corresponding policy via offline RL. Crucially, our proposed approach does not require actual physical rollouts or an accurate simulator for either the reward learning or policy optimization steps. To test our approach, we identify a subset of existing offline RL benchmarks that are well suited for offline reward learning and also propose new offline apprenticeship learning benchmarks which allow for more open-ended behaviors. Our empirical results suggest that combining offline RL with learned human preferences can enable an agent to learn to perform novel tasks that were not explicitly shown in the offline data.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 48 references that are in the map. First posted on arXiv 20 Jul 2021.

Builds on

34 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 31

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  10. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  11. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  12. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
19 more
  1. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  2. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  3. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  4. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  5. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  6. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  7. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  8. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  9. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  10. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  11. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  12. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
  13. 2020The MAGICAL Benchmark for Robust Imitationcited by 30
  14. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  15. 2019Truly Batch Apprenticeship Learning with Deep Successor Featurescited by 27
  16. 2017Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learningcited by 17
  17. 2018Risk-Sensitive Generative Adversarial Imitation Learningcited by 16
  18. 2019Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrationscited by 14
  19. 2020DERAIL: Diagnostic Environments for Reward And Imitation Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 7; 2025: 3.

17 papers.

Sort:

By area:

  1. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  3. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  4. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 63
  5. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  6. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  7. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  8. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  9. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  10. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  11. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  12. 2023STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularizationcited by 1
5 more
  1. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learningcited by 1
  2. 2025Offline Safe Policy Optimization From Heterogeneous Feedbackcited by 1
  3. 2024Online Policy Learning from Offline Preferencescited by 0
  4. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0
  5. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  2. 2024Preference Elicitation for Offline Reinforcement Learning
  3. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  4. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  5. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
  6. 2021A Minimalist Approach to Offline Reinforcement Learning
  7. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information
  8. 2023Boosting Offline Reinforcement Learning with Action Preference Query