mlsanityData to 7 Oct 2026

DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections

arXiv 1906.04733 ↗First posted 10 Jun 2019cs.LG cs.AI stat.MLcited by 254

Not on the map. Why?

Cited by 118 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In many real-world reinforcement learning applications, access to the environment is limited to a fixed dataset, instead of direct (online) interaction with the environment. When using this data for either evaluation or training of a new policy, accurate estimates of discounted stationary distribution ratios -- correction terms which quantify the likelihood that the new policy will experience a certain state-action pair normalized by the probability with which the state-action pair appears in the dataset -- can improve accuracy and performance. In this work, we propose an algorithm, DualDICE, for estimating these quantities. In contrast to previous approaches, our algorithm is agnostic to knowledge of the behavior policy (or policies) used to generate the dataset. Furthermore, it eschews any direct use of importance weights, thus avoiding potential optimization instabilities endemic of previous methods. In addition to providing theoretical guarantees, we present an empirical study of our algorithm applied to off-policy policy evaluation and find that our algorithm significantly improves accuracy compared to existing techniques.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 10; 2021: 53; 2022: 51; 2023: 54; 2024: 36; 2025: 28; 2026: 21.

254 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  4. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  5. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  6. 2021GFlowNet Foundationscited by 286
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  9. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  11. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  12. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
38 more
  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  2. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  3. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  4. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  5. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  6. 2021Reward is enough for convex MDPscited by 82
  7. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  8. 2021Off-Policy Imitation Learning from Observationscited by 70
  9. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  10. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  11. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  12. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  13. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  14. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  15. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  16. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  17. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  18. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  19. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  20. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  21. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  22. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  23. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  24. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  25. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  26. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  27. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  28. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  29. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  30. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  31. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  32. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  33. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  34. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  35. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  36. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  37. 2021Active Offline Policy Selectioncited by 26
  38. 2021Density Constrained Reinforcement Learningcited by 25

50 most cited of 254 All 254 (JSON)