# Distillation Guided Residual Learning for Binary Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2007.05223 · HTML: https://mlsanity.com/paper/2007.05223/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2020.

## Built on by (most cited first, 6 of 6)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [0/1 Deep Neural Networks via Block Coordinate Descent](https://mlsanity.com/paper/2206.09379/) (2022)
- [BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance](https://mlsanity.com/paper/2211.06987/) (2022)
- [Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer](https://mlsanity.com/paper/2104.00210/) (2021)
- [Frameless Graph Knowledge Distillation](https://mlsanity.com/paper/2307.06631/) (2023)

## Abstract (arXiv)

It is challenging to bridge the performance gap between Binary CNN (BCNN) and Floating point CNN (FCNN). We observe that, this performance gap leads to substantial residuals between intermediate feature maps of BCNN and FCNN. To minimize the performance gap, we enforce BCNN to produce similar intermediate feature maps with the ones of FCNN. This training strategy, i.e., optimizing each binary convolutional block with block-wise distillation loss derived from FCNN, leads to a more effective optimization to BCNN. It also motivates us to update the binary convolutional block architecture to facilitate the optimization of block-wise distillation loss. Specifically, a lightweight shortcut branch is inserted into each binary convolutional block to complement residuals at each block. Benefited from its Squeeze-and-Interaction (SI) structure, this shortcut branch introduces a fraction of parameters, e.g., 10% overheads, but effectively complements the residuals. Extensive experiments on ImageNet demonstrate the superior performance of our method in both classification efficiency and accuracy, e.g., BCNN trained with our methods achieves the accuracy of 60.45% on ImageNet.