# BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations

arXiv: https://arxiv.org/abs/2002.06517 · HTML: https://mlsanity.com/paper/2002.06517/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2020.

## Built on by (most cited first, 23 of 23)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [OMPQ: Orthogonal Mixed Precision Quantization](https://mlsanity.com/paper/2109.07865/) (2021)
- [Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning](https://mlsanity.com/paper/2109.01905/) (2021)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold](https://mlsanity.com/paper/2204.07439/) (2022)
- [Efficient Multitask Dense Predictor via Binarization](https://mlsanity.com/paper/2405.14136/) (2024)
- [Self-Distribution Binary Neural Networks](https://mlsanity.com/paper/2103.02394/) (2021)
- [BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements](https://mlsanity.com/paper/2110.08562/) (2021)
- [Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer](https://mlsanity.com/paper/2104.00210/) (2021)
- [CTMQ: Cyclic Training of Convolutional Neural Networks with Multiple Quantization Steps](https://mlsanity.com/paper/2206.12794/) (2022)
- [Ternary Quantization: A Survey](https://mlsanity.com/paper/2303.01505/) (2023)
- [Beyond Discreteness: Sample Complexity Analysis of Straight-Through Estimator for 1-bit Quantization](https://mlsanity.com/paper/2505.18113/) (2025)
- [RCT: Resource Constrained Training for Edge AI](https://mlsanity.com/paper/2103.14493/) (2021)
- [VC dimension of partially quantized neural networks in the overparametrized regime](https://mlsanity.com/paper/2110.02456/) (2021)
- [Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning](https://mlsanity.com/paper/2110.08851/) (2021)
- [PR-CIM: a Variation-Aware Binary-Neural-Network Framework for Process-Resilient Computation-in-memory](https://mlsanity.com/paper/2110.09962/) (2021)
- [Pixel to Binary Embedding Towards Robustness for CNNs](https://mlsanity.com/paper/2206.05898/) (2022)
- [IR2Net: Information Restriction and Information Recovery for Accurate Binary Neural Networks](https://mlsanity.com/paper/2210.02637/) (2022)
- [Designed Dithering Sign Activation for Binary Neural Networks](https://mlsanity.com/paper/2405.02220/) (2024)
- [Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks](https://mlsanity.com/paper/2601.22660/) (2026)
- [Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks](https://mlsanity.com/paper/2606.27759/) (2026)
- [A Target-Centric Survey of Quantization-Aware Training](https://mlsanity.com/paper/2608.29667/) (2026)

## Abstract (arXiv)

Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings. However, BNNs suffer from performance degradation mainly due to the gradient mismatch caused by binarizing activations. Previous works tried to address the gradient mismatch problem by reducing the discrepancy between activation functions used at forward pass and its differentiable approximation used at backward pass, which is an indirect measure. In this work, we use the gradient of smoothed loss function to better estimate the gradient mismatch in quantized neural network. Analysis using the gradient mismatch estimator indicates that using higher precision for activation is more effective than modifying the differentiable approximation of activation function. Based on the observation, we propose a new training scheme for binary activation networks called BinaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on various benchmarks with the same amount of parameters and computing cost.