mlsanityData to 7 Oct 2026

Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators

arXiv 2110.03549 ↗First posted 7 Oct 2021cs.LG cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

Discrete and especially binary random variables occur in many machine learning models, notably in variational autoencoders with binary latent states and in stochastic binary networks. When learning such models, a key tool is an estimator of the gradient of the expected loss with respect to the probabilities of binary variables. The straight-through (ST) estimator gained popularity due to its simplicity and efficiency, in particular in deep networks where unbiased estimators are impractical. Several techniques were proposed to improve over ST while keeping the same low computational complexity: Gumbel-Softmax, ST-Gumbel-Softmax, BayesBiNN, FouST. We conduct a theoretical analysis of bias and variance of these methods in order to understand tradeoffs and verify the originally claimed properties. The presented theoretical results allow for better understanding of these methods and in some cases reveal serious issues.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

27 of 37 references.

References · 27

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  5. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  6. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  7. 2014Neural Variational Inference and Learning in Belief Networkscited by 116
  8. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  9. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 104
  10. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 78
  11. 2016Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computingcited by 68
  12. 2013Deep AutoRegressive Networkscited by 60
15 more
  1. 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 36
  2. 2020High-Capacity Expert Binary Networkscited by 34
  3. 2017Learning Discrete Weights Using the Local Reparameterization Trickcited by 29
  4. 2019Back to Simplicity: How to Train Accurate BNNs from Scratch?cited by 24
  5. 2017Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resourcescited by 23
  6. 2019Improved training of binary networks for human pose estimation and image recognitioncited by 22
  7. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 22
  8. 2018Probabilistic Binary Neural Networkscited by 20
  9. 2020Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networkscited by 15
  10. 2014Techniques for Learning Binary Stochastic Feedforward Neural Networkscited by 14
  11. 2017Variational Deep Semantic Hashing for Text Documentscited by 12
  12. 2018NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashingcited by 11
  13. 2020Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimatorcited by 6
  14. 2020Path Sample-Analytic Gradient Estimators for Stochastic Binary Networkscited by 4
  15. 2020Self-Supervised Bernoulli Autoencoders for Semi-Supervised Hashingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1; 2026: 2.

4 papers.

  1. 2022AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networkscited by 2
  2. 2026Non-Myopic Active Feature Acquisition via Pathwise Policy Gradientscited by 1
  3. 2026DiffPrune: differentiable information throttling for token pruning in vision-language modelscited by 1
  4. 2025A Principled Bayesian Framework for Training Binary and Spiking Neural Networkscited by 0