# Bidirectional Model-based Policy Optimization

arXiv: https://arxiv.org/abs/2007.01995 · HTML: https://mlsanity.com/paper/2007.01995/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2020.

## Built on by (most cited first, 30 of 38)

- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2202.10324/) (2022)
- [Offline Reinforcement Learning with Reverse Model-based Imagination](https://mlsanity.com/paper/2110.00188/) (2021)
- [Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination](https://mlsanity.com/paper/2206.07989/) (2022)
- [Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size](https://mlsanity.com/paper/2211.11092/) (2022)
- [Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function](https://mlsanity.com/paper/2302.01244/) (2023)
- [Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption](https://mlsanity.com/paper/2405.19014/) (2024)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL](https://mlsanity.com/paper/2310.07220/) (2023)
- [On Effective Scheduling of Model-based Reinforcement Learning](https://mlsanity.com/paper/2111.08550/) (2021)
- [Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse](https://mlsanity.com/paper/2305.18443/) (2023)
- [Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning](https://mlsanity.com/paper/2405.17031/) (2024)
- [On Rollouts in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.16918/) (2025)
- [Dynamic Horizon Value Estimation for Model-based Reinforcement Learning](https://mlsanity.com/paper/2009.09593/) (2020)
- [EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model](https://mlsanity.com/paper/2210.00498/) (2022)
- [Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets](https://mlsanity.com/paper/2406.01150/) (2024)
- [Offline Imitation Learning with Model-based Reverse Augmentation](https://mlsanity.com/paper/2406.12550/) (2024)
- [Robust Imitation of a Few Demonstrations with a Backwards Model](https://mlsanity.com/paper/2210.09337/) (2022)
- [Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.09418/) (2022)
- [Model-based Reinforcement Learning with Multi-step Plan Value Estimation](https://mlsanity.com/paper/2209.05530/) (2022)
- [Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2303.03955/) (2023)
- [How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimization](https://mlsanity.com/paper/2309.12671/) (2023)
- [Grounded Answers for Multi-agent Decision-making Problem through Generative World Model](https://mlsanity.com/paper/2410.02664/) (2024)
- [Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2410.17373/) (2024)
- [Monte Carlo Tree Search in the Presence of Transition Uncertainty](https://mlsanity.com/paper/2312.11348/) (2023)
- [Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning](https://mlsanity.com/paper/2412.14312/) (2024)
- [Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rollouts](https://mlsanity.com/paper/2208.02434/) (2022)
- [Backward Learning for Goal-Conditioned Policies](https://mlsanity.com/paper/2312.05044/) (2023)

## Abstract (arXiv)

Model-based reinforcement learning approaches leverage a forward dynamics model to support planning and decision making, which, however, may fail catastrophically if the model is inaccurate. Although there are several existing methods dedicated to combating the model error, the potential of the single forward model is still limited. In this paper, we propose to additionally construct a backward dynamics model to reduce the reliance on accuracy in forward model predictions. We develop a novel method, called Bidirectional Model-based Policy Optimization (BMPO) to utilize both the forward model and backward model to generate short branched rollouts for policy optimization. Furthermore, we theoretically derive a tighter bound of return discrepancy, which shows the superiority of BMPO against the one using merely the forward model. Extensive experiments demonstrate that BMPO outperforms state-of-the-art model-based methods in terms of sample efficiency and asymptotic performance.