mlsanityData to 7 Oct 2026

COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation

arXiv 2204.08957 ↗First posted 19 Apr 2022cs.LG cs.AIcited by 69

Not on the map. Why?

Cited by 40 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset. This problem setting is appealing in many real-world scenarios, where direct interaction with the environment is costly or risky, and where the resulting policy should comply with safety constraints. However, it is challenging to compute a policy that guarantees satisfying the cost constraints in the offline RL setting, since the off-policy evaluation inherently has an estimation error. In this paper, we present an offline constrained RL algorithm that optimizes the policy in the space of the stationary distribution. Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for actual constraint satisfaction. Experimental results show that COptiDICE attains better policies in terms of constraint satisfaction and return-maximization, outperforming baseline algorithms.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 19 Apr 2022.

Builds on

35 of 51 references.

References · 35

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  10. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  11. 2018Reward Constrained Policy Optimizationcited by 397
  12. 2018Maximum a Posteriori Policy Optimisationcited by 318
23 more
  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2020Critic Regularized Regressioncited by 269
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  8. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  9. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  10. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  11. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  12. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  13. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  14. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  15. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  16. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  17. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  18. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  19. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  20. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  21. 2020Offline Policy Selection under Uncertaintycited by 31
  22. 2020Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecificationcited by 16
  23. 2022Settling the Communication Complexity for Distributed Offline Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 22; 2025: 19; 2026: 16.

69 papers.

Sort:

By area:

  1. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  2. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  3. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  4. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  5. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  6. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  7. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  8. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  9. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  10. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  11. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  12. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
38 more
  1. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  2. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
  3. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learningcited by 16
  4. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
  5. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learningcited by 13
  6. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  7. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  8. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  9. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  10. 2024Offline Safe Reinforcement Learning Using Trajectory Classificationcited by 9
  11. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  12. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
  13. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
  14. 2025V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functionscited by 5
  15. 2023Offline Diversity Maximization Under Imitation Constraintscited by 4
  16. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
  17. 2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcarecited by 4
  18. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  19. 2025Learning Conservative Neural Control Barrier Functions from Offline Datacited by 4
  20. 2023Tempo Adaptation in Non-stationary Reinforcement Learningcited by 3
  21. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  22. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  23. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learningcited by 3
  24. 2025From Uncertain to Safe: Conformal Adaptation of Diffusion Models for Safe PDE Controlcited by 3
  25. 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policiescited by 3
  26. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flowscited by 2
  27. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2
  28. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarioscited by 2
  29. 2025A Provable Approach for End-to-End Safe Reinforcement Learningcited by 2
  30. 2025Model-Based Diffusion Sampling for Predictive Control in Offline Decision Makingcited by 2
  31. 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learningcited by 2
  32. 2024Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learningcited by 1
  33. 2024Pre-Trained Vision Models as Perception Backbones for Safety Filters in Autonomous Drivingcited by 1
  34. 2025Offline Constrained Reinforcement Learning under Partial Data Coveragecited by 1
  35. 2025Semi-gradient DICE for Offline Constrained Reinforcement Learningcited by 1
  36. 2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
  37. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  38. 2025SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimationcited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
  2. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  3. 2023AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
  4. 2024A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs
  5. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  6. 2021A Minimalist Approach to Offline Reinforcement Learning
  7. 2025Policy Constraint by Only Support Constraint for Offline Reinforcement Learning
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory