mlsanityData to 14 Oct 2026

A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation

arXiv 2106.06854 ↗First posted 12 Jun 2021cs.LG cs.AI stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

Marginalized importance sampling (MIS), which measures the density ratio between the state-action occupancy of a target policy and that of a sampling distribution, is a promising approach for off-policy evaluation. However, current state-of-the-art MIS methods rely on complex optimization tricks and succeed mostly on simple toy problems. We bridge the gap between MIS and deep reinforcement learning by observing that the density ratio can be computed from the successor representation of the target policy. The successor representation can be trained through deep reinforcement learning methodology and decouples the reward optimization from the dynamics of the environment, making the resulting algorithm stable and applicable to high-dimensional domains. We evaluate the empirical performance of our approach on a variety of challenging Atari and MuJoCo environments.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 55 of its 81 references that are in the map. First posted on arXiv 12 Jun 2021.

Builds on

55 of 81 references.

References · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  11. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  12. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
38 more
  1. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  2. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  3. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  4. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  6. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  8. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  11. 2018State Representation Learning for Control: An Overviewcited by 127
  12. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  13. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  14. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  15. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  16. 2018Count-Based Exploration with the Successor Representationcited by 112
  17. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  18. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  19. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  20. 2016Off-policy evaluation for slate recommendationcited by 103
  21. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  22. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  23. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  24. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  25. 2016Deep Successor Reinforcement Learningcited by 72
  26. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  27. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  28. 2016Deep Reinforcement Learning with Successor Features for Navigation across Similar Environmentscited by 50
  29. 2018Horizon: Facebook's Open Source Applied Reinforcement Learning Platformcited by 48
  30. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  31. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  32. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  33. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  34. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 37
  35. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  36. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  37. 2019Deep Residual Reinforcement Learningcited by 21
  38. 2017Multi-step Off-policy Learning Without Importance Sampling Ratioscited by 19

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 5; 2025: 4; 2026: 2.

19 papers.

Sort:

By area:

  1. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  2. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 41
  3. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  4. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 25
  5. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  6. 2025Temporal Difference Flowscited by 24
  7. 2024A Distributional Analogue to the Successor Representationcited by 17
  8. 2022Data Representativity for Machine Learning and AI Systemscited by 15
  9. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 13
  10. 2023Marginal Density Ratio for Off-Policy Evaluation in Contextual Banditscited by 9
  11. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 9
  12. 2023Composing Task Knowledge with Modular Successor Feature Approximatorscited by 7
7 more
  1. 2024Predictive representations: building blocks of intelligencecited by 4
  2. 2024Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policiescited by 3
  3. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  4. 2025Average-DICE: Stationary Distribution Correction by Regressioncited by 2
  5. 2024Concept-driven Off Policy Evaluationcited by 1
  6. 2026Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settingscited by 0
  7. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0