mlsanityData to 7 Oct 2026

DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning

arXiv 2309.08925 ↗First posted 16 Sep 2023cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based reinforcement learning (RL), which learns an environment model from the offline dataset and generates more out-of-distribution model data, has become an effective approach to the problem of distribution shift in offline RL. Due to the gap between the learned and actual environment, conservatism should be incorporated into the algorithm to balance accurate offline data and imprecise model data. The conservatism of current algorithms mostly relies on model uncertainty estimation. However, uncertainty estimation is unreliable and leads to poor performance in certain scenarios, and the previous methods ignore differences between the model data, which brings great conservatism. To address the above issues, this paper proposes a milDly cOnservative Model-bAsed offlINe RL algorithm (DOMAIN) without estimating model uncertainty, and designs the adaptive sampling distribution of model samples, which can adaptively adjust the model data penalty. In this paper, we theoretically demonstrate that the Q value learned by the DOMAIN outside the region is a lower bound of the true Q value, the DOMAIN is less conservative than previous model-based offline RL algorithms, and has the guarantee of safety policy improvement. The results of extensive experiments show that DOMAIN outperforms prior RL algorithms and the average performance has improved by 1.8% on the D4RL benchmark.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 58 references that are in the map. First posted on arXiv 16 Sep 2023.

Builds on

41 of 58 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance4 Sep
  2. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  3. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  4. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  6. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  9. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning9 Nov
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning2 Oct
  12. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
8 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 21

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  2. 2016OpenAI Gymcited by 1,684
  3. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  11. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  12. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 136
9 more
  1. 2018Learning Scheduling Algorithms for Data Processing Clusterscited by 133
  2. 2018Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applicationscited by 122
  3. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  4. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  5. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  6. 2022A Survey on Model-based Reinforcement Learningcited by 65
  7. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  8. 2023CASOG: Conservative Actor-critic with SmOoth Gradient for Skill Learning in Robot-Assisted Interventioncited by 6
  9. 2020Policy Gradient from Demonstration and Curiositycited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  2. 2024Preference Adaptive and Sequential Text-to-Image Generationcited by 8
  3. 2024Offline Model-Based Reinforcement Learning with Anti-Explorationcited by 1
  4. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  2. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  3. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  5. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
  6. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  7. 2023A Survey on Offline Model-Based Reinforcement Learning
  8. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage