mlsanityData to 7 Oct 2026

In-Distribution Imagination for Model-Based Offline Reinforcement Learning

arXiv 2609.38673 ↗First posted 29 Sep 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms also fits: Diffusion and flow policies for offline RL

Abstract (arXiv)

Model-based offline reinforcement learning (MBORL) improves sample efficiency through model-generated trajectories. However, accumulative model error can drive imagined trajectories outside the offline data distribution, leading to unrealistic synthetic data and unstable policy optimization. Many existing methods primarily control rollouts using transition-level uncertainty. We propose in-distribution imagination (IDI), a rollout control framework that estimates trajectory support in a learned representation space and adaptively truncates rollouts that leave the offline trajectory manifold. Combined with trajectory-regularized RL, an extension of entropy-regularized RL, IDI consistently improves performance in limited-data settings. Experiments show that trajectory support predicts rollout failure substantially better than transition-level uncertainty, highlighting the importance of trajectory-level rollout control in MBORL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 19 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

18 of 19 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning23 Aug
  2. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 12

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  4. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  5. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  6. 2024Learning-based Trajectory Tracking for Bird-inspired Flapping-Wing Robotscited by 5
  7. 2025Estimation of Aerodynamics Forces in Dynamic Morphing Wing Flightcited by 4
  8. 2025RoVerFly: Robust and Versatile Implicit Hybrid Control of Quadrotor-Payload Systemscited by 4
  9. 2026Finite Memory Belief Approximation for Optimal Control in Partially Observable Markov Decision Processescited by 4
  10. 2026WOMBET: World Model-Based Experience Transfer for Robust and Sample-efficient Reinforcement Learningcited by 2
  11. 2026Robust Adversarial Policy Optimization Under Dynamics Uncertaintycited by 2
  12. 2026Trajectory-Regularized Stochastic Optimal Control via KL Divergencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On Training in Imagination
  2. 2025Imagination-Limited Q-Learning for Offline Reinforcement Learning
  3. 2021Offline Reinforcement Learning with Reverse Model-based Imagination
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  5. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  6. 2026MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning
  7. 2025Offline Reinforcement Learning with Generative Trajectory Policies
  8. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization