# Accurate and Compact Convolutional Neural Networks with Trained Binarization

arXiv: https://arxiv.org/abs/1909.11366 · HTML: https://mlsanity.com/paper/1909.11366/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2019.

## Built on by (most cited first, 20 of 20)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [MART: MultiscAle Relational Transformer Networks for Multi-agent Trajectory Prediction](https://mlsanity.com/paper/2407.21635/) (2024)
- ["BNN - BN = ?": Training Binary Neural Networks without Batch Normalization](https://mlsanity.com/paper/2104.08215/) (2021)
- [BinaryViT: Pushing Binary Vision Transformers Towards Convolutional Models](https://mlsanity.com/paper/2306.16678/) (2023)
- [Lightweight Pixel Difference Networks for Efficient Visual Representation Learning](https://mlsanity.com/paper/2402.00422/) (2024)
- [Binarized Low-light Raw Video Enhancement](https://mlsanity.com/paper/2403.19944/) (2024)
- [0/1 Deep Neural Networks via Block Coordinate Descent](https://mlsanity.com/paper/2206.09379/) (2022)
- [Custom Gradient Estimators are Straight-Through Estimators in Disguise](https://mlsanity.com/paper/2405.05171/) (2024)
- [A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network](https://mlsanity.com/paper/2403.03739/) (2024)
- [AdaptCL: Adaptive Continual Learning for Tackling Heterogeneity in Sequential Datasets](https://mlsanity.com/paper/2207.11005/) (2022)
- [Learning in Order! A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2409.04473/) (2024)
- [Croesus: Multi-Stage Processing and Transactions for Video-Analytics in Edge-Cloud Systems](https://mlsanity.com/paper/2201.00063/) (2021)
- [S3NN: Time Step Reduction of Spiking Surrogate Gradients for Training Energy Efficient Single-Step Spiking Neural Networks](https://mlsanity.com/paper/2201.10879/) (2022)
- [RedBit: An End-to-End Flexible Framework for Evaluating the Accuracy of Quantized CNNs](https://mlsanity.com/paper/2301.06193/) (2023)
- [HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks](https://mlsanity.com/paper/2205.09839/) (2022)
- [Temporal Flow Mask Attention for Open-Set Long-Tailed Recognition of Wild Animals in Camera-Trap Images](https://mlsanity.com/paper/2208.14625/) (2022)
- [ARBiBench: Benchmarking Adversarial Robustness of Binarized Neural Networks](https://mlsanity.com/paper/2312.13575/) (2023)
- [IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models](https://mlsanity.com/paper/2404.13504/) (2024)
- [Binarized High-Efficiency RAW Video Restoration and Beyond](https://mlsanity.com/paper/2608.16756/) (2026)

## Abstract (arXiv)

Although convolutional neural networks (CNNs) are now widely used in various computer vision applications, its huge resource demanding on parameter storage and computation makes the deployment on mobile and embedded devices difficult. Recently, binary convolutional neural networks are explored to help alleviate this issue by quantizing both weights and activations with only 1 single bit. However, there may exist a noticeable accuracy degradation when compared with full-precision models. In this paper, we propose an improved training approach towards compact binary CNNs with higher accuracy. Trainable scaling factors for both weights and activations are introduced to increase the value range. These scaling factors will be trained jointly with other parameters via backpropagation. Besides, a specific training algorithm is developed including tight approximation for derivative of discontinuous binarization function and L2 regularization acting on weight scaling factors. With these improvements, the binary CNN achieves 92.3% accuracy on CIFAR-10 with VGG-Small network. On ImageNet, our method also obtains 46.1% top-1 accuracy with AlexNet and 54.2% with Resnet-18 surpassing previous works.