mlsanityData to 7 Oct 2026

GenDICE: Generalized Offline Estimation of Stationary Values

arXiv 2002.09072 ↗First posted 21 Feb 2020stat.ML cs.LGcited by 122

Not on the map. Why?

Cited by 68 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this challenging scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove its consistency under general conditions, provide an error analysis, and demonstrate strong empirical performance on benchmark problems, including off-line PageRank and off-policy policy evaluation.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 6; 2021: 32; 2022: 25; 2023: 25; 2024: 14; 2025: 10; 2026: 10.

122 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  8. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  9. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  10. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  11. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  12. 2021Off-Policy Imitation Learning from Observationscited by 70
38 more
  1. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  2. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  3. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  4. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  5. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  6. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  7. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  8. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  9. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  10. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  11. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  12. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  13. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  14. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  15. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  16. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  17. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  18. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  19. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  20. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  21. 2021Flexible Option Learningcited by 24
  22. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  23. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  24. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  25. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  26. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  27. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  28. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  29. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  30. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  31. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  32. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 17
  33. 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
  34. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  35. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  36. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settingscited by 14
  37. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  38. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14

50 most cited of 122 All 122 (JSON)