mlsanityData to 7 Oct 2026

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning

arXiv 2206.07166 ↗First posted 14 Jun 2022cs.LG cs.AIcited by 14

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) extends the paradigm of classical RL algorithms to purely learning from static datasets, without interacting with the underlying environment during the learning process. A key challenge of offline RL is the instability of policy training, caused by the mismatch between the distribution of the offline data and the undiscounted stationary state-action distribution of the learned policy. To avoid the detrimental impact of distribution mismatch, we regularize the undiscounted stationary distribution of the current policy towards the offline data during the policy optimization process. Further, we train a dynamics model to both implement this regularization and better estimate the stationary distribution of the current policy, reducing the error induced by distribution mismatch. On a wide range of continuous-control offline RL datasets, our method indicates competitive performance, which validates our algorithm. The code is publicly available.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 61 of its 99 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

61 of 99 references.

Closest ancestors · 19

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  6. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  7. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  10. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  11. 2020Critic Regularized Regression26 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
7 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  6. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2017Improved Training of Wasserstein GANscited by 2,223
  7. 2014Conditional Generative Adversarial Netscited by 2,152
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2018Demystifying MMD GANscited by 1,268
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
30 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  7. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  8. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  12. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  13. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  14. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  15. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  16. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  17. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  18. 2018Offline A/B testing for Recommender Systemscited by 94
  19. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  20. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  21. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  22. 2020Bayesian Attention Modulescited by 51
  23. 2019Learning When-to-Treat Policiescited by 49
  24. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  25. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  26. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  27. 2022ALLSH: Active Learning Guided by Local Sensitivity and Hardnesscited by 23
  28. 2020Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributionscited by 22
  29. 2020Implicit Distributional Reinforcement Learningcited by 16
  30. 2021Alignment Attention by Matching Key and Query Distributionscited by 14

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 7; 2024: 4; 2025: 2.

14 papers.

Sort:

By area:

  1. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  2. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  3. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  4. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  5. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  6. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  7. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  8. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  9. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  10. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  11. 2023A Survey of Demonstration Learningcited by 12
  12. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
2 more
  1. 2025Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devicescited by 2
  2. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  2. 2022Offline Policy Optimization in RL with Variance Regularizaton
  3. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  4. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  5. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  6. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  7. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  8. 2024Integrating Domain Knowledge for handling Limited Data in Offline RL