# Credit Assignment Techniques in Stochastic Computation Graphs

arXiv: https://arxiv.org/abs/1901.01761 · HTML: https://mlsanity.com/paper/1901.01761/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2019.

## Built on by (most cited first, 14 of 14)

- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [Sparse Graph Learning from Spatiotemporal Time Series](https://mlsanity.com/paper/2205.13492/) (2022)
- [Reparameterized Policy Learning for Multimodal Trajectory Optimization](https://mlsanity.com/paper/2307.10710/) (2023)
- [Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis](https://mlsanity.com/paper/2306.16803/) (2023)
- [A unified view of likelihood ratio and reparameterization gradients](https://mlsanity.com/paper/2105.14900/) (2021)
- [Storchastic: A Framework for General Stochastic Automatic Differentiation](https://mlsanity.com/paper/2104.00428/) (2021)
- [Probabilistic Programming with Programmable Variational Inference](https://mlsanity.com/paper/2406.15742/) (2024)
- [Cognitive Models and AI Algorithms Provide Templates for Designing Language Agents](https://mlsanity.com/paper/2602.22523/) (2026)
- [Data-Driven Reinforcement Learning for Virtual Character Animation Control](https://mlsanity.com/paper/2104.06358/) (2021)
- [Hindsight Network Credit Assignment: Efficient Credit Assignment in Networks of Discrete Stochastic Units](https://mlsanity.com/paper/2110.07700/) (2021)
- [Variance Reduction for Score Functions Using Optimal Baselines](https://mlsanity.com/paper/2212.13587/) (2022)
- [GradInf: Gradient Estimation as Probabilistic Inference](https://mlsanity.com/paper/2607.07840/) (2026)
- [Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting](https://mlsanity.com/paper/2609.12890/) (2026)
- [Backward-State Policy Is Part of the Learning Algorithm](https://mlsanity.com/paper/2609.39813/) (2026)

## Abstract (arXiv)

Stochastic computation graphs (SCGs) provide a formalism to represent structured optimization problems arising in artificial intelligence, including supervised, unsupervised, and reinforcement learning. Previous work has shown that an unbiased estimator of the gradient of the expected loss of SCGs can be derived from a single principle. However, this estimator often has high variance and requires a full model evaluation per data point, making this algorithm costly in large graphs. In this work, we address these problems by generalizing concepts from the reinforcement learning literature. We introduce the concepts of value functions, baselines and critics for arbitrary SCGs, and show how to use them to derive lower-variance gradient estimates from partial model evaluations, paving the way towards general and efficient credit assignment for gradient-based optimization. In doing so, we demonstrate how our results unify recent advances in the probabilistic inference and reinforcement learning literature.