mlsanityData to 7 Oct 2026

Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination

arXiv 2206.07989 ↗First posted 16 Jun 2022cs.LG cs.AIcited by 21

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The learned policy of model-free offline reinforcement learning (RL) methods is often constrained to stay within the support of datasets to avoid possible dangerous out-of-distribution actions or states, making it challenging to handle out-of-support region. Model-based RL methods offer a richer dataset and benefit generalization by generating imaginary trajectories with either trained forward or reverse dynamics model. However, the imagined transitions may be inaccurate, thus downgrading the performance of the underlying offline RL method. In this paper, we propose to augment the offline dataset by using trained bidirectional dynamics models and rollout policies with double check. We introduce conservatism by trusting samples that the forward model and backward model agree on. Our method, confidence-aware bidirectional offline model-based imagination, generates reliable samples and can be combined with any model-free offline RL method. Experimental results on the D4RL benchmarks demonstrate that our method significantly boosts the performance of existing model-free offline RL algorithms and achieves competitive or better scores against baseline methods.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 86 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

58 of 86 references.

Closest ancestors · 23

Same direction, newest first

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  2. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  3. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  6. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  7. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  8. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  11. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  12. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
11 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  4. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  5. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 35

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2016OpenAI Gymcited by 1,684
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
23 more
  1. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  3. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  4. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  5. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  6. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  7. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  8. 2020Model-Based Offline Planningcited by 126
  9. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  10. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  11. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  12. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  13. 2019Exploring Model-based Planning with Policy Networkscited by 90
  14. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  15. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  16. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  17. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  18. 2020Bidirectional Model-based Policy Optimizationcited by 38
  19. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  20. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  21. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  22. 2018Forward-Backward Reinforcement Learningcited by 17
  23. 2018Towards a Simple Approach to Multi-step Model-based Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 6; 2026: 1.

21 papers.

Sort:

By area:

  1. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
  2. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  3. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  4. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  5. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  6. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  7. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  8. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learningcited by 12
  9. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimizationcited by 6
  10. 2024Improving Offline Reinforcement Learning with Inaccurate Simulatorscited by 4
  11. 2023CROP: Conservative Reward for Model-based Offline Policy Optimizationcited by 3
  12. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generationcited by 2
9 more
  1. 2025Offline Reinforcement Learning via Inverse Optimizationcited by 2
  2. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  3. 2023HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approachcited by 1
  4. 2025Extendable Planning via Multiscale Diffusioncited by 1
  5. 2025Exploration by Random Distribution Distillationcited by 1
  6. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learningcited by 0
  7. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  8. 2025RAD: Retrieval High-quality Demonstrations to Enhance Decision-makingcited by 0
  9. 2026In-Distribution Imagination for Model-Based Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fully Offline Reinforcement Learning
  2. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  4. 2025A Clean Slate for Offline Reinforcement Learning
  5. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
  6. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  7. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  8. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning