mlsanityData to 14 Oct 2026

GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values

arXiv 2001.11113 ↗First posted 29 Jan 2020cs.LG stat.MLcited by 70

Not on the map. Why?

Cited by 26 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present GradientDICE for estimating the density ratio between the state distribution of the target policy and the sampling distribution in off-policy reinforcement learning. GradientDICE fixes several problems of GenDICE (Zhang et al., 2020), the state-of-the-art for estimating such density ratios. Namely, the optimization problem in GenDICE is not a convex-concave saddle-point problem once nonlinearity in optimization variable parameterization is introduced to ensure positivity, so any primal-dual algorithm is not guaranteed to converge or find the desired solution. However, such nonlinearity is essential to ensure the consistency of GenDICE even with a tabular representation. This is a fundamental contradiction, resulting from GenDICE's original formulation of the optimization problem. In GradientDICE, we optimize a different objective from GenDICE by using the Perron-Frobenius theorem and eliminating GenDICE's use of divergence. Consequently, nonlinearity in parameterization is not necessary for GradientDICE, which is provably convergent under linear function approximation.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 14; 2022: 16; 2023: 20; 2024: 9; 2025: 6; 2026: 3.

70 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  5. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  6. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  7. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learningcited by 42
  8. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  9. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  10. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  11. 2023Contrastive Difference Predictive Codingcited by 36
  12. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
38 more
  1. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 31
  2. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  3. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  4. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  5. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 23
  6. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 21
  7. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  8. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  9. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  10. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  11. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
  12. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  13. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  14. 2022User-Interactive Offline Reinforcement Learningcited by 11
  15. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  16. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
  17. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximationcited by 10
  18. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learningcited by 9
  19. 2022Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstractioncited by 7
  20. 2023Revisiting a Design Choice in Gradient Temporal Difference Learningcited by 7
  21. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  22. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  23. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  24. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  25. 2023Model-based Offline Reinforcement Learning with Local Misspecificationcited by 5
  26. 2023Marginalized Importance Sampling for Off-Environment Policy Evaluationcited by 5
  27. 2024Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Featurescited by 5
  28. 2025Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Featurescited by 5
  29. 2023Offline Diversity Maximization Under Imitation Constraintscited by 4
  30. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4
  31. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspectivecited by 4
  32. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  33. 2025Curiosity Meets Cooperation: A Game-Theoretic Approach to Long-Tail Multi-Label Learningcited by 4
  34. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approachcited by 3
  35. 2023Probabilistic Offline Policy Ranking with Approximate Bayesian Computationcited by 3
  36. 2024Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policiescited by 3
  37. 2022Stochastic Gradient Descent with Dependent Data for Offline Reinforcement Learningcited by 2
  38. 2023Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structurecited by 2

50 most cited of 70 All 70 (JSON)