mlsanityData to 14 Oct 2026

AlgaeDICE: Policy Gradient from Arbitrary Experience

arXiv 1912.02074 ↗First posted 4 Dec 2019cs.LG cs.AIcited by 188

Not on the map. Why?

Cited by 122 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.

Key facts. Cited by 188 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 8; 2021: 39; 2022: 46; 2023: 38; 2024: 36; 2025: 14; 2026: 7.

188 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  6. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
38 more
  1. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  2. 2021Offline RL Without Off-Policy Evaluationcited by 151
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  4. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  5. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  6. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  7. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  8. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  9. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  10. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  11. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  12. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  13. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  14. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  15. 2021Off-Policy Imitation Learning from Observationscited by 70
  16. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  17. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  18. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  19. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 61
  20. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  21. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
  22. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  23. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  24. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  25. 2023Behavior Proximal Policy Optimizationcited by 43
  26. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  27. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  28. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  29. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 39
  30. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  31. 2023Boosting Continuous Control with Consistency Policycited by 38
  32. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  33. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 36
  34. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  35. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  36. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  37. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 31
  38. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29

50 most cited of 188 All 188 (JSON)