mlsanityData to 14 Oct 2026

Generative OOD-regularized Model-based Policy Optimization

arXiv 2605.24405 ↗First posted 23 May 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

We study sequential decision-making with offline reinforcement learning (RL). Traditional offline RL policies may result in out-of-distribution (OOD) actions when training relies only on sparse offline representations. To ensure safe offline policies in a sparse state-action space, we explore how density estimation models can be integrated into model-based RL methods to avoid the OOD regions. Generative models are capable of explicitly modeling the density in sparse state-action spaces. Building on this, we introduce Generative OOD-regularized Model-based Policy Optimization (GORMPO), a density-regularized offline RL algorithm that uses generative density modeling to restrict policy updates to high-density areas of the dataset. Furthermore, we examine whether better OOD detection corresponds to better model-based offline policies. We compare (1) the OOD detection capabilities of various density estimators and (2) their performance within the GORMPO framework on a real-world medical dataset and sparse offline RL datasets. We theoretically guarantee GORMPO's performance under mild assumptions. Empirically, GORMPO outperforms state-of-the-art baselines by 17% on a real-world medical dataset and enhances the base model on the offline RL datasets. Our empirical findings show that better OOD detection generally results in improved policies in environments with stable dynamics, while conservative penalties with poor density estimation are favored when dynamics are uncertain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 50 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

37 of 50 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learning23 Aug
  2. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning30 Jun
  3. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning28 Jan
  4. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  5. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  12. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 25

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2018Neural Ordinary Differential Equationscited by 3,779
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  9. 2017Billion-scale similarity search with GPUscited by 2,497
  10. 2015Variational Inference with Normalizing Flowscited by 2,170
  11. 2016Density estimation using Real NVPcited by 2,042
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
13 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  4. 2025Mean Flows for One-step Generative Modelingcited by 561
  5. 2015Importance Weighted Autoencoderscited by 456
  6. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 425
  7. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 397
  8. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  9. 2016Deep Structured Energy Based Models for Anomaly Detectioncited by 114
  10. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  11. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  12. 2025Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategiescited by 4
  13. 2025Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devicescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions
  2. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
  3. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
  4. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  5. 2026Offline Policy Optimization with Posterior Sampling
  6. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
  7. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  8. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL