mlsanityData to 14 Oct 2026

A unified view of likelihood ratio and reparameterization gradients

arXiv 2105.14900 ↗First posted 31 May 2021cs.LG cs.AI stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used to estimate gradients of expectations throughout machine learning and reinforcement learning; however, they are usually explained as simple mathematical tricks, with no insight into their nature. We use a first principles approach to explain that LR and RP are alternative methods of keeping track of the movement of probability mass, and the two are connected via the divergence theorem. Moreover, we show that the space of all possible estimators combining LR and RP can be completely parameterized by a flow field u(x) and an importance sampling distribution q(x). We prove that there cannot exist a single-sample estimator of this type outside our characterized space, thus, clarifying where we should be searching for better Monte Carlo gradient estimators.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 41 of its 68 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

41 of 68 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  7. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  11. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  12. 2015Gradient Estimation Using Stochastic Computation Graphscited by 121
29 more
  1. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  2. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 106
  3. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  4. 2015Hierarchical Variational Modelscited by 89
  5. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85
  6. 2018Implicit Reparameterization Gradientscited by 85
  7. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 81
  8. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 79
  9. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  10. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  11. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  12. 2018Variance reduction properties of the reparameterization trickcited by 41
  13. 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
  14. 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 37
  15. 2018Pathwise Derivatives Beyond the Reparameterization Trickcited by 36
  16. 2016The Generalized Reparameterization Gradientcited by 30
  17. 2018Using Large Ensembles of Control Variates for Variational Inferencecited by 21
  18. 2017Expected Policy Gradientscited by 16
  19. 2019Credit Assignment Techniques in Stochastic Computation Graphscited by 14
  20. 2019Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learningcited by 12
  21. 2018Reparameterization Gradient for Non-differentiable Modelscited by 11
  22. 2016Overdispersed Black-Box Variational Inferencecited by 8
  23. 2017Mean Actor Criticcited by 8
  24. 2019Total stochastic gradient algorithms and applications in reinforcement learningcited by 7
  25. 2019GO Gradient for Expectation-Based Objectivescited by 5
  26. 2018Pathwise Derivatives for Multivariate Distributionscited by 3
  27. 2019A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling schemecited by 2
  28. 2019New Tricks for Estimating Gradients of Expectationscited by 1
  29. 2019Generalized Transformation-based Gradientcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 4; 2025: 0; 2026: 4.

12 papers.

  1. 2022Tutorial on amortized optimizationcited by 83
  2. 2022Gradient Estimation with Discrete Stein Operatorscited by 23
  3. 2022Image-based Treatment Effect Heterogeneitycited by 20
  4. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 18
  5. 2024Fast and Unified Path Gradient Estimators for Normalizing Flowscited by 5
  6. 2024Revisiting Experience Replayable Conditionscited by 3
  7. 2024Mollification Effects of Policy Gradient Methodscited by 2
  8. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  9. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 2
  10. 2026Retry Policy Gradients in Continuous Action Spacescited by 1
  11. 2023Machine learning using magnetic stochastic synapsescited by 0
  12. 2026Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?cited by 0