mlsanityData to 7 Oct 2026

A Unified Framework for Alternating Offline Model Training and Policy Learning

arXiv 2210.05922 ↗First posted 12 Oct 2022cs.LG cs.AIcited by 14

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline model-based reinforcement learning (offline MBRL), we learn a dynamic model from historically collected data, and subsequently utilize the learned model and fixed datasets for policy learning, without further interacting with the environment. Offline MBRL algorithms can improve the efficiency and stability of policy learning over the model-free algorithms. However, in most of the existing offline MBRL algorithms, the learning objectives for the dynamic models and the policies are isolated from each other. Such an objective mismatch may lead to inferior performance of the learned agents. In this paper, we address this issue by developing an iterative offline MBRL framework, where we maximize a lower bound of the true expected return, by alternating between dynamic-model training and policy learning. With the proposed unified model-policy learning framework, we achieve competitive performance on a wide range of continuous-control offline reinforcement learning datasets. Source code is publicly released.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 68 of its 105 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

68 of 105 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning14 Jun
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  4. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  5. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  10. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
4 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
38 more
  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  2. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  3. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  4. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  7. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  8. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  11. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  12. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  13. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  14. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  15. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  16. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  17. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  18. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  19. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  20. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  21. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  22. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  23. 2020Bayesian Attention Modulescited by 51
  24. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  25. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  26. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  27. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  28. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  29. 2021Bayesian Attention Belief Networkscited by 25
  30. 2022ALLSH: Active Learning Guided by Local Sensitivity and Hardnesscited by 23
  31. 2020Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributionscited by 22
  32. 2018Representation Balancing MDPs for Off-Policy Policy Evaluationcited by 20
  33. 2019Gradient-Aware Model-based Policy Searchcited by 19
  34. 2020Batch Stationary Distribution Estimationcited by 19
  35. 2020Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learningcited by 16
  36. 2020Implicit Distributional Reinforcement Learningcited by 16
  37. 2021Minimax Model Learningcited by 16
  38. 2021Alignment Attention by Matching Key and Query Distributionscited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 3; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  2. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  3. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  4. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  5. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  6. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  7. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  8. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  9. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  10. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  11. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  12. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learningcited by 0
2 more
  1. 2024SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimationcited by 0
  2. 2026Are Good Generators Good Decision-Makers? Policy Learning for General Interventions via Retargeted Counterfactual Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Offline Model-Based Reinforcement Learning
  2. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  3. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model
  4. 2020Model-Based Offline Planning
  5. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning
  6. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  7. 2021Offline RL Without Off-Policy Evaluation
  8. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning