mlsanityData to 7 Oct 2026

OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation

arXiv 2106.10783 ↗First posted 21 Jun 2021cs.LG cs.AIcited by 121

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider the offline reinforcement learning (RL) setting where the agent aims to optimize the policy solely from the data without further environment interactions. In offline RL, the distributional shift becomes the primary source of difficulty, which arises from the deviation of the target policy being optimized from the behavior policy used for data collection. This typically causes overestimation of action values, which poses severe problems for model-free algorithms that use bootstrapping. To mitigate the problem, prior offline RL algorithms often used sophisticated techniques that encourage underestimation of action values, which introduces an additional set of hyperparameters that need to be tuned properly. In this paper, we present an offline RL algorithm that prevents overestimation in a more principled way. Our algorithm, OptiDICE, directly estimates the stationary distribution corrections of the optimal policy and does not rely on policy-gradients, unlike previous offline RL algorithms. Using an extensive set of benchmark datasets for offline RL, we show that OptiDICE performs competitively with the state-of-the-art methods.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 46 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

29 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2021Convex optimizationcited by 2,479
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
11 more
  1. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  2. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  3. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  4. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  5. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  6. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  7. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  8. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  9. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  10. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  11. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 15; 2023: 24; 2024: 28; 2025: 24; 2026: 27.

121 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 162
  2. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  4. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  5. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  6. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  7. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  8. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  9. 2025Horizon Reduction Makes RL Scalablecited by 55
  10. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  11. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  12. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
38 more
  1. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  2. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  3. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  4. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  5. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  6. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  7. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  8. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  9. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  10. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  11. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  12. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  13. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 17
  14. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  15. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  16. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  17. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraintscited by 15
  18. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  19. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  20. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  21. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  22. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 13
  23. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
  24. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
  25. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 11
  26. 2021Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correctioncited by 10
  27. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimationcited by 10
  28. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  29. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  30. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  31. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  32. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  33. 2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
  34. 2024Offline Safe Reinforcement Learning Using Trajectory Classificationcited by 9
  35. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  36. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  37. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  38. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2021Offline RL Without Off-Policy Evaluation
  3. 2021The Least Restriction for Offline Reinforcement Learning
  4. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  5. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  6. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation
  7. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  8. 2020Hyperparameter Selection for Offline Reinforcement Learning