mlsanityData to 14 Oct 2026

Model-Based Offline Planning with Trajectory Pruning

arXiv 2105.07351 ↗First posted 16 May 2021cs.AI cs.SY eess.SYcited by 30

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The recent offline reinforcement learning (RL) studies have achieved much progress to make RL usable in real-world systems by learning policies from pre-collected datasets without environment interaction. Unfortunately, existing offline RL methods still face many practical challenges in real-world system control tasks, such as computational restriction during agent training and the requirement of extra control flexibility. The model-based planning framework provides an attractive alternative. However, most model-based planning algorithms are not designed for offline settings. Simply combining the ingredients of offline RL with existing methods either provides over-restrictive planning or leads to inferior performance. We propose a new light-weighted model-based offline planning framework, namely MOPP, which tackles the dilemma between the restrictions of offline learning and high-performance planning. MOPP encourages more aggressive trajectory rollout guided by the behavior policy learned from data, and prunes out problematic trajectories to avoid potential out-of-distribution samples. Experimental results show that MOPP provides competitive performance compared with existing model-based offline planning and RL approaches.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 35 references that are in the map. First posted on arXiv 16 May 2021.

Builds on

24 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 307
  10. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  11. 2019Batch Policy Learning under Constraintscited by 244
  12. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
5 more
  1. 2020Model-Based Offline Planningcited by 129
  2. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
  3. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  4. 2019Exploring Model-based Planning with Policy Networkscited by 90
  5. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 9; 2024: 6; 2025: 5; 2026: 1.

30 papers.

Sort:

By area:

  1. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  2. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  3. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  4. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 43
  5. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  6. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  7. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  8. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  9. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  10. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  11. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  12. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
18 more
  1. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  2. 2025Closing the Train-Test Gap in World Models for Gradient-Based Planningcited by 11
  3. 2022Model-based trajectory stitching for improved behavioural cloning and its applicationscited by 9
  4. 2023Offline Multi-Agent Reinforcement Learning with Coupled Value Factorizationcited by 7
  5. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  6. 2023Model-based Offline Reinforcement Learning with Local Misspecificationcited by 5
  7. 2023Get Back Here: Robust Imitation by Return-to-Distribution Planningcited by 4
  8. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  9. 2025Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lenscited by 4
  10. 2022Behavior Estimation from Multi-Source Data for Offline Reinforcement Learningcited by 2
  11. 2023Multi-timestep models for Model-based Reinforcement Learningcited by 1
  12. 2024Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learningcited by 1
  13. 2024Offline Trajectory Optimization for Offline Reinforcement Learningcited by 1
  14. 2024State-Constrained Offline Reinforcement Learningcited by 1
  15. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  16. 2026Receding-Horizon Control via Drifting Modelscited by 1
  17. 2024M3PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Modelcited by 0
  18. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Offline Model-Based Reinforcement Learning
  2. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  3. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  5. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  6. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  7. 2026Shortcut Trajectory Planning for Efficient Offline Reinforcement Learning
  8. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning