# AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks

arXiv: https://arxiv.org/abs/2112.02880 · HTML: https://mlsanity.com/paper/2112.02880/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 27 of 27)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://mlsanity.com/paper/1903.05662/) (2019)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [fastai: A Layered API for Deep Learning](https://mlsanity.com/paper/2002.04688/) (2020)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Adversarial Robustness vs Model Compression, or Both?](https://mlsanity.com/paper/1903.12561/) (2019)
- [Rotated Binary Neural Network](https://mlsanity.com/paper/2009.13055/) (2020)
- [A Survey on Methods and Theories of Quantized Neural Networks](https://mlsanity.com/paper/1808.04752/) (2018)
- [Model Compression with Adversarial Robustness: A Unified Optimization Framework](https://mlsanity.com/paper/1902.03538/) (2019)
- [Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization](https://mlsanity.com/paper/1906.02107/) (2019)
- [ProxQuant: Quantized Neural Networks via Proximal Operators](https://mlsanity.com/paper/1810.00861/) (2018)
- [Training Binary Neural Networks through Learning with Noisy Supervision](https://mlsanity.com/paper/2010.04871/) (2020)
- [Training Binary Neural Networks using the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10778/) (2020)
- [Mirror Descent View for Neural Network Quantization](https://mlsanity.com/paper/1910.08237/) (2019)
- [Improving Adversarial Robustness in Weight-quantized Neural Networks](https://mlsanity.com/paper/2012.14965/) (2020)
- [Contrastive Learning for Lifted Networks](https://mlsanity.com/paper/1905.02507/) (2019)
- [Proximal Mean-field for Neural Network Quantization](https://mlsanity.com/paper/1812.04353/) (2018)

## Built on by (most cited first, 11 of 11)

- [Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning](https://mlsanity.com/paper/2302.11147/) (2023)
- [BiPer: Binary Neural Networks using a Periodic Function](https://mlsanity.com/paper/2404.01278/) (2024)
- [CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training](https://mlsanity.com/paper/2510.18784/) (2025)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [Binarized Simplicial Convolutional Neural Networks](https://mlsanity.com/paper/2405.04098/) (2024)
- [Binarizing by Classification: Is soft function really necessary?](https://mlsanity.com/paper/2205.07433/) (2022)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Learning-driven Physically-aware Large-scale Circuit Gate Sizing](https://mlsanity.com/paper/2403.08193/) (2024)
- [Miniaturized Graph Convolutional Networks with Topologically Consistent Pruning](https://mlsanity.com/paper/2306.17590/) (2023)
- [Long-Tailed Recognition on Binary Networks by Calibrating A Pre-trained Model](https://mlsanity.com/paper/2404.00285/) (2024)
- [End-to-end fully-binarized network design: from Generic Learned Thermometer to Block Pruning](https://mlsanity.com/paper/2505.13462/) (2025)

## Abstract (arXiv)

We propose a new algorithm for training deep neural networks (DNNs) with binary weights. In particular, we first cast the problem of training binary neural networks (BiNNs) as a bilevel optimization instance and subsequently construct flexible relaxations of this bilevel program. The resulting training method shares its algorithmic simplicity with several existing approaches to train BiNNs, in particular with the straight-through gradient estimator successfully employed in BinaryConnect and subsequent methods. In fact, our proposed method can be interpreted as an adaptive variant of the original straight-through estimator that conditionally (but not always) acts like a linear mapping in the backward pass of error propagation. Experimental results demonstrate that our new algorithm offers favorable performance compared to existing approaches.