mlsanityData to 7 Oct 2026

AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks

arXiv 2112.02880 ↗First posted 6 Dec 2021cs.LG cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

We propose a new algorithm for training deep neural networks (DNNs) with binary weights. In particular, we first cast the problem of training binary neural networks (BiNNs) as a bilevel optimization instance and subsequently construct flexible relaxations of this bilevel program. The resulting training method shares its algorithmic simplicity with several existing approaches to train BiNNs, in particular with the straight-through gradient estimator successfully employed in BinaryConnect and subsequent methods. In fact, our proposed method can be interpreted as an adaptive variant of the original straight-through estimator that conditionally (but not always) acts like a linear mapping in the backward pass of error propagation. Experimental results demonstrate that our new algorithm offers favorable performance compared to existing approaches.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

27 of 52 references.

References · 27

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  6. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  7. 2019Learned Step Size Quantizationcited by 604
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  10. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 251
  11. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  12. 2020Binary Neural Networks: A Surveycited by 184
15 more
  1. 2020fastai: A Layered API for Deep Learningcited by 183
  2. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  3. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  4. 2021The Bayesian Learning Rulecited by 95
  5. 2020Rotated Binary Neural Networkcited by 77
  6. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  7. 2019Model Compression with Adversarial Robustness: A Unified Optimization Frameworkcited by 71
  8. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 58
  9. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 56
  10. 2020Training Binary Neural Networks through Learning with Noisy Supervisioncited by 39
  11. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 22
  12. 2019Mirror Descent View for Neural Network Quantizationcited by 15
  13. 2020Improving Adversarial Robustness in Weight-quantized Neural Networkscited by 11
  14. 2019Contrastive Learning for Lifted Networkscited by 9
  15. 2018Proximal Mean-field for Neural Network Quantizationcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 4; 2025: 2.

11 papers.

  1. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  2. 2024BiPer: Binary Neural Networks using a Periodic Functioncited by 9
  3. 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Trainingcited by 4
  4. 2023Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Trainingcited by 3
  5. 2024Binarized Simplicial Convolutional Neural Networkscited by 3
  6. 2022Binarizing by Classification: Is soft function really necessary?cited by 2
  7. 2022AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networkscited by 2
  8. 2023Miniaturized Graph Convolutional Networks with Topologically Consistent Pruningcited by 0
  9. 2024Learning-driven Physically-aware Large-scale Circuit Gate Sizingcited by 0
  10. 2024Long-Tailed Recognition on Binary Networks by Calibrating A Pre-trained Modelcited by 0
  11. 2025End-to-end fully-binarized network design: from Generic Learned Thermometer to Block Pruningcited by 0