mlsanityData to 7 Oct 2026

Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL

arXiv 2106.09119 ↗First posted 16 Jun 2021cs.LGcited by 24

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) aims to extract near-optimal policies from imperfect offline data without additional environment interactions. Extracting policies from diverse offline datasets has the potential to expand the range of applicability of RL by making the training process safer, faster, and more streamlined. We investigate how to improve the performance of offline RL algorithms, its robustness to the quality of offline data, as well as its generalization capabilities. To this end, we introduce Offline Model-based RL with Adaptive Behavioral Priors (MABE). Our algorithm is based on the finding that dynamics models, which support within-domain generalization, and behavioral priors, which support cross-domain generalization, are complementary. When combined together, they substantially improve the performance and generalization of offline RL policies. In the widely studied D4RL offline RL benchmark, we find that MABE achieves higher average performance compared to prior model-free and model-based algorithms. In experiments that require cross-domain generalization, we find that MABE outperforms prior methods. Our website is available at https://sites.google.com/berkeley.edu/mabe .

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 64 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

39 of 64 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 29

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
17 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2019Solving Rubik's Cube with a Robot Handcited by 699
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  7. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  8. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  9. 2018To Trust Or Not To Trust A Classifiercited by 205
  10. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  11. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  12. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  13. 2020Model-Based Offline Planningcited by 126
  14. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  15. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  16. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  17. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 4; 2024: 6; 2025: 1; 2026: 1.

24 papers.

Sort:

By area:

  1. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  4. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  5. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  6. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 18
  7. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  8. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  9. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  10. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
  11. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  12. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
12 more
  1. 2024Efficient Imitation Learning with Conservative World Modelscited by 11
  2. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  3. 2022ASPiRe:Adaptive Skill Priors for Reinforcement Learningcited by 7
  4. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 6
  5. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
  6. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  7. 2021Offline RL With Resource Constrained Online Deploymentcited by 1
  8. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  9. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learningcited by 0
  10. 2023Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planningcited by 0
  11. 2024SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimationcited by 0
  12. 2026Offline Multi-Agent Reinforcement Learning with a Physics-Informed World Model for Cooperative Mixed Traffic Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  2. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  3. 2024Integrating Domain Knowledge for handling Limited Data in Offline RL
  4. 2025A Clean Slate for Offline Reinforcement Learning
  5. 2023The Generalization Gap in Offline Reinforcement Learning
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  7. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretraining
  8. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization