# Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators

arXiv: https://arxiv.org/abs/2110.03549 · HTML: https://mlsanity.com/paper/2110.03549/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 27 of 27)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models](https://mlsanity.com/paper/1703.07370/) (2017)
- [Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/1603.08270/) (2016)
- [Deep AutoRegressive Networks](https://mlsanity.com/paper/1310.8499/) (2013)
- [MuProp: Unbiased Backpropagation for Stochastic Neural Networks](https://mlsanity.com/paper/1511.05176/) (2015)
- [High-Capacity Expert Binary Networks](https://mlsanity.com/paper/2010.03558/) (2020)
- [Learning Discrete Weights Using the Local Reparameterization Trick](https://mlsanity.com/paper/1710.07739/) (2017)
- [Back to Simplicity: How to Train Accurate BNNs from Scratch?](https://mlsanity.com/paper/1906.08637/) (2019)
- [Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resources](https://mlsanity.com/paper/1703.00862/) (2017)
- [Training Binary Neural Networks using the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10778/) (2020)
- [Improved training of binary networks for human pose estimation and image recognition](https://mlsanity.com/paper/1904.05868/) (2019)
- [Probabilistic Binary Neural Networks](https://mlsanity.com/paper/1809.03368/) (2018)
- [Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks](https://mlsanity.com/paper/2006.06880/) (2020)
- [Techniques for Learning Binary Stochastic Feedforward Neural Networks](https://mlsanity.com/paper/1406.2989/) (2014)
- [Variational Deep Semantic Hashing for Text Documents](https://mlsanity.com/paper/1708.03436/) (2017)
- [NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing](https://mlsanity.com/paper/1805.05361/) (2018)
- [Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimator](https://mlsanity.com/paper/2005.10477/) (2020)
- [Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks](https://mlsanity.com/paper/2006.03143/) (2020)
- [Self-Supervised Bernoulli Autoencoders for Semi-Supervised Hashing](https://mlsanity.com/paper/2007.08799/) (2020)

## Built on by (most cited first, 4 of 4)

- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Non-Myopic Active Feature Acquisition via Pathwise Policy Gradients](https://mlsanity.com/paper/2605.05511/) (2026)
- [DiffPrune: differentiable information throttling for token pruning in vision-language models](https://mlsanity.com/paper/2608.01985/) (2026)
- [A Principled Bayesian Framework for Training Binary and Spiking Neural Networks](https://mlsanity.com/paper/2505.17962/) (2025)

## Abstract (arXiv)

Discrete and especially binary random variables occur in many machine learning models, notably in variational autoencoders with binary latent states and in stochastic binary networks. When learning such models, a key tool is an estimator of the gradient of the expected loss with respect to the probabilities of binary variables. The straight-through (ST) estimator gained popularity due to its simplicity and efficiency, in particular in deep networks where unbiased estimators are impractical. Several techniques were proposed to improve over ST while keeping the same low computational complexity: Gumbel-Softmax, ST-Gumbel-Softmax, BayesBiNN, FouST. We conduct a theoretical analysis of bias and variance of these methods in order to understand tradeoffs and verify the originally claimed properties. The presented theoretical results allow for better understanding of these methods and in some cases reveal serious issues.