mlsanityData to 14 Oct 2026

Backpropagation through the Void: Optimizing control variates for black-box gradient estimation

arXiv 1711.00123 ↗First posted 31 Oct 2017cs.LGcited by 106

Not on the map. Why?

Abstract (arXiv)

Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables. Our method uses gradients of a neural network trained jointly with model parameters or policies, and is applicable in both discrete and continuous settings. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 2; 2020: 0; 2021: 28; 2022: 22; 2023: 20; 2024: 15; 2025: 8; 2026: 11.

106 papers.

Sort:

By area:

  1. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  2. 2022GFlowNets and variational inferencecited by 106
  3. 2021Implicit MLE: Backpropagating Through Discrete Exponential Family Distributionscited by 98
  4. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  5. 2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
  6. 2022Ordered Subgraph Aggregation Networkscited by 57
  7. 2023Learning to Maximize Mutual Information for Dynamic Feature Selectioncited by 56
  8. 2021Smooth Normalizing Flowscited by 53
  9. 2021Variational Causal Networks: Approximate Bayesian Inference over Causal Structurescited by 43
  10. 2021RieszNet and ForestRiesz: Automatic Debiased Machine Learning with Neural Nets and Random Forestscited by 42
  11. 2022SIMPLE: A Gradient Estimator for k-Subset Samplingcited by 34
  12. 2023Bridging Discrete and Backpropagation: Straight-Through and Beyondcited by 31
38 more
  1. 2022Automatic Differentiation of Programs with Discrete Randomnesscited by 29
  2. 2021Direct Differentiable Augmentation Searchcited by 28
  3. 2021Multi-Facet Clustering Variational Autoencoderscited by 28
  4. 2023Neural Probabilistic Logic Programming in Discrete-Continuous Domainscited by 25
  5. 2021Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Searchcited by 24
  6. 2022Gradient Estimation with Discrete Stein Operatorscited by 23
  7. 2023Efficient Attention via Control Variatescited by 23
  8. 2021CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signalscited by 22
  9. 2022Sparse Graph Learning from Spatiotemporal Time Seriescited by 22
  10. 2021Self-Conditioned Probabilistic Learning of Video Rescalingcited by 19
  11. 2022SABLAS: Learning Safe Control for Black-box Dynamical Systemscited by 17
  12. 2023Sparse Backpropagation for MoE Trainingcited by 17
  13. 2021Learning Generalized Gumbel-max Causal Mechanismscited by 16
  14. 2024Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomycited by 16
  15. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  16. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  17. 2021An overview of mixing augmentation methods and augmentation strategiescited by 14
  18. 2022Neural Set Function Extensions: Learning with Discrete Functions in High Dimensionscited by 14
  19. 2023Differentiable Sampling of Categorical Distributions Using the CatLog-Derivative Trickcited by 14
  20. 2021A unified view of likelihood ratio and reparameterization gradientscited by 12
  21. 2022A Survey of Automated Data Augmentation Algorithms for Deep Learning-based Image Classification Taskscited by 12
  22. 2022Adaptive Perturbation-Based Gradient Estimation for Discrete Latent Variable Modelscited by 12
  23. 2025Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithmscited by 12
  24. 2021Storchastic: A Framework for General Stochastic Automatic Differentiationcited by 11
  25. 2023GeoPhy: Differentiable Phylogenetic Inference via Geometric Gradients of Tree Topologiescited by 10
  26. 2021Model-free Policy Learning with Reward Gradientscited by 9
  27. 2021ARMS: Antithetic-REINFORCE-Multi-Sample Gradient for Binary Variablescited by 9
  28. 2021Coupled Gradient Estimators for Discrete Latent Variablescited by 9
  29. 2021Double Control Variates for Gradient Estimation in Discrete Latent Variable Modelscited by 9
  30. 2021Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPscited by 8
  31. 2021Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spacescited by 7
  32. 2022Gradient Estimation for Binary Latent Variables via Gradient Variance Clippingcited by 7
  33. 2023Meta-learning Control Variates: Variance Reduction with Limited Datacited by 7
  34. 2022Deep invariant networks with differentiable augmentation layerscited by 6
  35. 2023SLACK: Stable Learning of Augmentations with Cold-start and KL regularizationcited by 6
  36. 2024Variance reduction of diffusion model's gradients with Taylor approximation-based control variatecited by 6
  37. 2019Hierarchical Variational Imitation Learning of Control Programscited by 5
  38. 2022Variational Open-Domain Question Answeringcited by 5

50 most cited of 106 All 106 (JSON)