mlsanityData to 14 Oct 2026

Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation

arXiv 2106.13125 ↗First posted 24 Jun 2021cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

35 of 50 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2013Auto-Encoding Variational Bayescited by 4,053
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  9. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  10. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  11. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  12. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
23 more
  1. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  2. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  3. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  4. 2018Meta-Gradient Reinforcement Learningcited by 159
  5. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  6. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  7. 2015Gradient Estimation Using Stochastic Computation Graphscited by 121
  8. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  9. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  10. 2018Evolved Policy Gradientscited by 88
  11. 2020Discovering Reinforcement Learning Algorithmscited by 77
  12. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  13. 2019Meta-learning of Sequential Strategiescited by 70
  14. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 68
  15. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  16. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  17. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  18. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 44
  19. 2020Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learningcited by 28
  20. 2019Adaptive Trade-Offs in Off-Policy Learningcited by 20
  21. 2019From Importance Sampling to Doubly Robust Policy Gradientcited by 17
  22. 2019Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learningcited by 12
  23. 2020Taylor Expansion Policy Optimizationcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 0; 2024: 0; 2025: 1.

5 papers.

  1. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  2. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 12
  3. 2021Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learningcited by 5
  4. 2022An Investigation of the Bias-Variance Tradeoff in Meta-Gradientscited by 5
  5. 2022Reinforcement Learning with Brain-Inspired Modulation can Improve Adaptation to Environmental Changescited by 0