mlsanityData to 7 Oct 2026

Bidirectional Model-based Policy Optimization

arXiv 2007.01995 ↗First posted 4 Jul 2020cs.LG cs.AI stat.MLcited by 38

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning approaches leverage a forward dynamics model to support planning and decision making, which, however, may fail catastrophically if the model is inaccurate. Although there are several existing methods dedicated to combating the model error, the potential of the single forward model is still limited. In this paper, we propose to additionally construct a backward dynamics model to reduce the reliance on accuracy in forward model predictions. We develop a novel method, called Bidirectional Model-based Policy Optimization (BMPO) to utilize both the forward model and backward model to generate short branched rollouts for policy optimization. Furthermore, we theoretically derive a tighter bound of return discrepancy, which shows the superiority of BMPO against the one using merely the forward model. Extensive experiments demonstrate that BMPO outperforms state-of-the-art model-based methods in terms of sample efficiency and asymptotic performance.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2020: 1; 2021: 5; 2022: 11; 2023: 9; 2024: 10; 2025: 2.

38 papers.

Sort:

By area:

  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  2. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  3. 2022A Survey on Model-based Reinforcement Learningcited by 65
  4. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  5. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  6. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  7. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  8. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  9. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  10. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  11. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  12. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
26 more
  1. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  2. 2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learningcited by 10
  3. 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
  4. 2020Dynamic Horizon Value Estimation for Model-based Reinforcement Learningcited by 8
  5. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8
  6. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  7. 2024Offline Imitation Learning with Model-based Reverse Augmentationcited by 8
  8. 2022Robust Imitation of a Few Demonstrations with a Backwards Modelcited by 7
  9. 2022Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learningcited by 6
  10. 2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
  11. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  12. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  13. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
  14. 2024Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learningcited by 4
  15. 2024Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learningcited by 4
  16. 2023Monte Carlo Tree Search in the Presence of Transition Uncertaintycited by 3
  17. 2022Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rolloutscited by 2
  18. 2023Backward Learning for Goal-Conditioned Policiescited by 2
  19. 2024Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learningcited by 2
  20. 2023Beware of Instantaneous Dependence in Reinforcement Learningcited by 1
  21. 2021ED2: Environment Dynamics Decomposition World Models for Continuous Controlcited by 0
  22. 2022Stock Trading Optimization through Model-based Reinforcement Learning with Resistance Support Relative Strengthcited by 0
  23. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  24. 2024Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learningcited by 0
  25. 2024Diminishing Return of Value Expansion Methodscited by 0
  26. 2025A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimizationcited by 0