Data to 7 Oct 2026

Monte Carlo Gradient Estimation in Machine Learning

arXiv 1906.10652 ↗First posted 25 Jun 2019stat.ML cs.LG math.OCcited by 281

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper is a broad and accessible survey of the methods we have at our disposal for Monte Carlo gradient estimation in machine learning and across the statistical sciences: the problem of computing the gradient of an expectation of a function with respect to parameters defining the distribution that is integrated; the problem of sensitivity analysis. In machine learning research, this gradient problem lies at the core of many learning problems, in supervised, unsupervised and reinforcement learning. We will generally seek to rewrite such gradients in a form that allows for Monte Carlo estimation, allowing them to be easily and efficiently used and analysed. We explore three strategies--the pathwise, score function, and measure-valued gradient estimators--exploring their historical development, derivation, and underlying assumptions. We describe their use in other fields, show how they are related and can be combined, and expand on their possible generalisations. Wherever Monte Carlo gradient estimators have been derived and deployed in the past, important advances have followed. A deeper and more widely-held understanding of this problem will lead to further advances, and it is these advances that we wish to support.

Key facts. Cited by 281 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 43; 2023: 52; 2024: 52; 2025: 55; 2026: 37.

281 papers.

Sort:

By area:

2023Training Diffusion Models with Reinforcement Learningcited by 939
2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
2021Neural Temporal Point Processes: A Reviewcited by 113
2021DiBS: Differentiable Bayesian Structure Learningcited by 105
2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
2022Tutorial on amortized optimizationcited by 83
2022Do Differentiable Simulators Give Better Policy Gradients?cited by 83
2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
2021Gradients are Not All You Needcited by 82
2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81

38 more

2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
2022Gradients without Backpropagationcited by 73
2021BCD Nets: Scalable Variational Approaches for Bayesian Causal Discoverycited by 68
2024Variational Bayesian Last Layerscited by 67
2021Risk-Averse Offline Reinforcement Learningcited by 65
2022A Survey on Model-based Reinforcement Learningcited by 65
2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
2022Constrained Policy Optimization via Bayesian World Modelscited by 55
2022A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inferencecited by 50
2021Variational Neural Annealingcited by 45
2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 40
2021Challenges and Opportunities in High-dimensional Variational Inferencecited by 35
2022Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterizationcited by 35
2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
2021Order Matters: Probabilistic Modeling of Node Sequence for Graph Generationcited by 32
2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 29
2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
2022Automatic Differentiation of Programs with Discrete Randomnesscited by 28
2021Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Boundscited by 26
2021Pathfinder: Parallel quasi-Newton variational inferencecited by 26
2023Graph Deep Learning for Time Series Forecastingcited by 26
2022Gradient Estimation with Discrete Stein Operatorscited by 22
2022An Introduction to Quantum Machine Learning for Engineerscited by 22
2022Sparse Graph Learning from Spatiotemporal Time Seriescited by 22
2021Autoregressive neural-network wavefunctions for ab initio quantum chemistrycited by 21
2022A Differential Entropy Estimator for Training Neural Networkscited by 20
2024On the limits of agency in agent-based modelscited by 20
2025Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posteriorcited by 20
2025Guiding Generative Models to Uncover Diverse and Novel Crystals via Reinforcement Learningcited by 19
2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulationcited by 18
2022Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selectioncited by 17
2023Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Boxcited by 17
2024PWM: Policy Learning with Multi-Task World Modelscited by 17
2021Ab-initio study of interacting fermions at finite temperature with neural canonical transformationcited by 16
2022Amortized Variational Inference: A Systematic Reviewcited by 16
2021Generalized Doubly Reparameterized Gradient Estimatorscited by 14
2021Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learningcited by 14

50 most cited of 281 All 281 (JSON)