mlsanityData to 14 Oct 2026

SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation

arXiv 2412.06486 ↗First posted 9 Dec 2024cs.LG cs.AIcited by 0

On the map: World models → Latent dynamics models and model-based RL

also in 2 more
Abstract (arXiv)

In offline reinforcement learning, deriving an effective policy from a pre-collected set of experiences is challenging due to the distribution mismatch between the target policy and the behavioral policy used to collect the data, as well as the limited sample size. Model-based reinforcement learning improves sample efficiency by generating simulated experiences using a learned dynamic model of the environment. However, these synthetic experiences often suffer from the same distribution mismatch. To address these challenges, we introduce SimuDICE, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model. SimuDICE enhances the quality of these simulated experiences by adjusting the sampling probabilities of state-action pairs based on stationary DIstribution Correction Estimation (DICE) and the estimated confidence in the model's predictions. This approach guides policy improvement by balancing experiences similar to those frequently encountered with ones that have a distribution mismatch. Our experiments show that SimuDICE achieves performance comparable to existing algorithms while requiring fewer pre-collected experiences and planning steps, and it remains robust across varying data collection policies.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 27 of its 39 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

27 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Diffusion for World Modeling: Visual Details Matter in Atari20 May
  2. 2023Mastering Diverse Domains through World Models10 Jan
  3. 2020Mastering Atari with Discrete World Models5 Oct
  4. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  5. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb

Other references · 22

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  8. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  12. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
10 more
  1. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
  2. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 96
  3. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  4. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  5. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  6. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 43
  7. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 36
  8. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 28
  9. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLcited by 24
  10. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024COSBO: Conservative Offline Simulation-Based Policy Optimization
  2. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  3. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  5. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  6. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  7. 2026Offline Policy Optimization with Posterior Sampling
  8. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model