How to train accurate BNNs for embedded systems?
Not on the map. Why?
Abstract (arXiv)
A key enabler of deploying convolutional neural networks on resource-constrained embedded systems is the binary neural network (BNN). BNNs save on memory and simplify computation by binarizing both features and weights. Unfortunately, binarization is inevitably accompanied by a severe decrease in accuracy. To reduce the accuracy gap between binary and full-precision networks, many repair methods have been proposed in the recent past, which we have classified and put into a single overview in this chapter. The repair methods are divided into two main branches, training techniques and network topology changes, which can further be split into smaller categories. The latter category introduces additional cost (energy consumption or additional area) for an embedded system, while the former does not. From our overview, we observe that progress has been made in reducing the accuracy gap, but BNN papers are not aligned on what repair methods should be used to get highly accurate BNNs. Therefore, this chapter contains an empirical review that evaluates the benefits of many repair methods in isolation over the ResNet-20&CIFAR10 and ResNet-18&CIFAR100 benchmarks. We found three repair categories most beneficial: feature binarizer, feature normalization, and double residual. Based on this review we discuss future directions and research opportunities. We sketch the benefit and costs associated with BNNs on embedded systems because it remains to be seen whether BNNs will be able to close the accuracy gap while staying highly energy-efficient on resource-constrained embedded systems.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 24 Jun 2022.
Builds on
41 of 59 references.
References · 41
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
- 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
- 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
- 2020Binary Neural Networks: A Surveycited by 184
- 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
- 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
- 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
- 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
- 2019XNOR-Net++: Improved Binary Neural Networkscited by 94
29 more
- 2021Network Quantization with Element-wise Gradient Scalingcited by 82
- 2020Rotated Binary Neural Networkcited by 77
- 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 58
- 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
- 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 51
- 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
- 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
- 2021A comprehensive review of Binary Neural Networkcited by 44
- 2020Training Binary Neural Networks through Learning with Noisy Supervisioncited by 39
- 2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 38
- 2018Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentationcited by 37
- 2020High-Capacity Expert Binary Networkscited by 34
- 2019Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagationcited by 28
- 2019Improved training of binary networks for human pose estimation and image recognitioncited by 22
- 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
- 2020Controlling Information Capacity of Binary Neural Networkcited by 15
- 2019Balanced Binary Neural Networks with Gated Residualcited by 14
- 2018Regularized Binary Network Trainingcited by 13
- 2021Dynamic Binary Neural Network by learning channel-wise thresholdscited by 13
- 2021Equal Bits: Enforcing Equally Distributed Binary Network Weightscited by 9
- 2020CUTIE: Beyond PetaOp/s/W Ternary DNN Inference Acceleration with Better-than-Binary Energy Efficiencycited by 8
- 2020ChewBaccaNN: A Flexible 223 TOPS/W BNN Acceleratorcited by 7
- 2021Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networkscited by 7
- 2020BCNN: A Binary CNN with All Matrix Ops Quantized to 1 Bit Precisioncited by 6
- 2021BoolNet: Minimizing The Energy Consumption of Binary Neural Networkscited by 6
- 2021Self-Distribution Binary Neural Networkscited by 4
- 2021A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networkscited by 3
- 2021Improving Binary Neural Networks through Fully Utilizing Latent Weightscited by 3
- 2021BCNN: Binary Complex Neural Networkcited by 2
Built on by
Citing papers per year: 2023: 1.
1 paper.