mlsanityData to 7 Oct 2026

SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasets

arXiv 2406.09486 ↗First posted 13 Jun 2024cs.CV cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based offline reinforcement Learning (RL) is a promising approach that leverages existing data effectively in many real-world applications, especially those involving high-dimensional inputs like images and videos. To alleviate the distribution shift issue in offline RL, existing model-based methods heavily rely on the uncertainty of learned dynamics. However, the model uncertainty estimation becomes significantly biased when observations contain complex distractors with non-trivial dynamics. To address this challenge, we propose a new approach - Separated Model-based Offline Policy Optimization (SeMOPO) - decomposing latent states into endogenous and exogenous parts via conservative sampling and estimating model uncertainty on the endogenous states only. We provide a theoretical guarantee of model uncertainty and performance bound of SeMOPO. To assess the efficacy, we construct the Low-Quality Vision Deep Data-Driven Datasets for RL (LQV-D4RL), where the data are collected by non-expert policy and the observations include moving distractors. Experimental results show that our method substantially outperforms all baseline methods, and further analytical experiments validate the critical designs in our method. The project website is hrefhttps://sites.google.com/view/semopohttps://sites.google.com/view/semopo.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 67 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

50 of 67 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  2. 2023Offline Reinforcement Learning with Imbalanced Datasets6 Jul
  3. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  4. 2022Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling16 Dec
  5. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  11. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  12. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
4 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 34

  1. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  2. 2017The Kinetics Human Action Video Datasetcited by 1,894
  3. 2016OpenAI Gymcited by 1,684
  4. 2018World Modelscited by 1,444
  5. 2020Mastering Atari with Discrete World Modelscited by 990
  6. 2018DeepMind Control Suitecited by 935
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  12. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
22 more
  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  2. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  3. 2019Self-Supervised Exploration via Disagreementcited by 263
  4. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  5. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  6. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  7. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  8. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  9. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 96
  10. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  11. 2021Learning Task Informed Abstractionscited by 58
  12. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  13. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  14. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  15. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
  16. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
  17. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  18. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  19. 2022Sample-Efficient Reinforcement Learning in the Presence of Exogenous Informationcited by 26
  20. 2023Learning World Models with Identifiable Factorizationcited by 23
  21. 2022Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learningcited by 21
  22. 2023Ignorance is Bliss: Robust Control via Information Gatingcited by 10

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8

Similar, not cited

Close in wording, no citation either way

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization
  2. 2024Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
  3. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think
  4. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions
  5. 2026Generative OOD-regularized Model-based Policy Optimization
  6. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations
  7. 2023Adversarial Model for Offline Reinforcement Learning
  8. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning