# Gradient Estimation Using Stochastic Computation Graphs

arXiv: https://arxiv.org/abs/1506.05254 · HTML: https://mlsanity.com/paper/1506.05254/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2015.

## Built on by (most cited first, 30 of 121)

- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://mlsanity.com/paper/2103.02438/) (2021)
- [Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions](https://mlsanity.com/paper/2106.01798/) (2021)
- [Soft Merging of Experts with Adaptive Routing](https://mlsanity.com/paper/2306.03745/) (2023)
- [Do Differentiable Simulators Give Better Policy Gradients?](https://mlsanity.com/paper/2202.00817/) (2022)
- [Gradients are Not All You Need](https://mlsanity.com/paper/2111.05803/) (2021)
- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [C2N: Practical Generative Noise Modeling for Real-World Denoising](https://mlsanity.com/paper/2202.09533/) (2022)
- [ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing](https://mlsanity.com/paper/2412.14711/) (2024)
- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inference](https://mlsanity.com/paper/2212.12393/) (2022)
- [A Variational Perspective on Generative Flow Networks](https://mlsanity.com/paper/2210.07992/) (2022)
- [Automatic Differentiation of Programs with Discrete Randomness](https://mlsanity.com/paper/2210.08572/) (2022)
- [On a few pitfalls in KL divergence gradient estimation for RL](https://mlsanity.com/paper/2506.09477/) (2025)
- [Learning Belief Representations for Imitation Learning in POMDPs](https://mlsanity.com/paper/1906.09510/) (2019)
- [Generalizable Human Pose Triangulation](https://mlsanity.com/paper/2110.00280/) (2021)
- [Gradient Estimation with Discrete Stein Operators](https://mlsanity.com/paper/2202.09497/) (2022)
- [Parallel Attention Network with Sequence Matching for Video Grounding](https://mlsanity.com/paper/2105.08481/) (2021)
- [CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signals](https://mlsanity.com/paper/2106.13695/) (2021)
- [Sparse Graph Learning from Spatiotemporal Time Series](https://mlsanity.com/paper/2205.13492/) (2022)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation](https://mlsanity.com/paper/2405.17784/) (2024)
- [Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation](https://mlsanity.com/paper/2412.12089/) (2024)
- [CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study](https://mlsanity.com/paper/2307.11346/) (2023)
- [Emergent Communication under Competition](https://mlsanity.com/paper/2101.10276/) (2021)
- [Differentiable Particle Filtering without Modifying the Forward Pass](https://mlsanity.com/paper/2106.10314/) (2021)
- [PWM: Policy Learning with Multi-Task World Models](https://mlsanity.com/paper/2407.02466/) (2024)
- [Nested Variational Inference](https://mlsanity.com/paper/2106.11302/) (2021)
- [Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis](https://mlsanity.com/paper/2306.16803/) (2023)
- [Neurosymbolic Diffusion Models](https://mlsanity.com/paper/2505.13138/) (2025)

## Abstract (arXiv)

In a variety of problems originating in supervised, unsupervised, and reinforcement learning, the loss function is defined by an expectation over a collection of random variables, which might be part of a probabilistic model or the external world. Estimating the gradient of this loss function, using samples, lies at the core of gradient-based learning algorithms for these problems. We introduce the formalism of stochastic computation graphs---directed acyclic graphs that include both deterministic functions and conditional probability distributions---and describe how to easily and automatically derive an unbiased estimator of the loss function's gradient. The resulting algorithm for computing the gradient estimator is a simple modification of the standard backpropagation algorithm. The generic scheme we propose unifies estimators derived in variety of prior work, along with variance-reduction techniques therein. It could assist researchers in developing intricate models involving a combination of stochastic and deterministic operations, enabling, for example, attention, memory, and control actions.