mlsanityData to 7 Oct 2026

Domain Generalization for Robust Model-Based Offline Reinforcement Learning

arXiv 2211.14827 ↗First posted 27 Nov 2022cs.LG cs.AI stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Existing offline reinforcement learning (RL) algorithms typically assume that training data is either: 1) generated by a known policy, or 2) of entirely unknown origin. We consider multi-demonstrator offline RL, a middle ground where we know which demonstrators generated each dataset, but make no assumptions about the underlying policies of the demonstrators. This is the most natural setting when collecting data from multiple human operators, yet remains unexplored. Since different demonstrators induce different data distributions, we show that this can be naturally framed as a domain generalization problem, with each demonstrator corresponding to a different domain. Specifically, we propose Domain-Invariant Model-based Offline RL (DIMORL), where we apply Risk Extrapolation (REx) (Krueger et al., 2020) to the process of learning dynamics and rewards models. Our results show that models trained with REx exhibit improved domain generalization performance when compared with the natural baseline of pooling all demonstrators' data. We observe that the resulting models frequently enable the learning of superior policies in the offline model-based RL setting, can improve the stability of the policy learning process, and potentially enable increased exploration.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 31 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

27 of 31 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  2. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2019Invariant Risk Minimizationcited by 1,832
  3. 2016OpenAI Gymcited by 1,684
  4. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  8. 2021Domain Generalization: A Surveycited by 760
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  11. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  12. 2020Rescaling Egocentric Visioncited by 529
6 more
  1. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  2. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  3. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  4. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  5. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Integrating Domain Knowledge for handling Limited Data in Offline RL
  2. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
  3. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
  4. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  5. 2025Statistical Guarantees for Offline Domain Randomization
  6. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correction
  7. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts
  8. 2023The Generalization Gap in Offline Reinforcement Learning