# PokeBNN: A Binary Pursuit of Lightweight Accuracy

arXiv: https://arxiv.org/abs/2112.00133 · HTML: https://mlsanity.com/paper/2112.00133/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [XNOR-Net++: Improved Binary Neural Networks](https://mlsanity.com/paper/1909.13863/) (2019)
- [Training Quantized Nets: A Deeper Understanding](https://mlsanity.com/paper/1706.02379/) (2017)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations](https://mlsanity.com/paper/2012.12206/) (2020)
- [Bitwise Neural Networks](https://mlsanity.com/paper/1601.06071/) (2016)
- [Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation](https://mlsanity.com/paper/1811.12755/) (2018)
- [Pareto-Optimal Quantized ResNet Is Mostly 4-bit](https://mlsanity.com/paper/2105.03536/) (2021)
- [Improved training of binary networks for human pose estimation and image recognition](https://mlsanity.com/paper/1904.05868/) (2019)
- [Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization](https://mlsanity.com/paper/1708.01001/) (2017)

## Built on by (most cited first, 30 of 49)

- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation](https://mlsanity.com/paper/2506.07530/) (2025)
- [BitNet b1.58 2B4T Technical Report](https://mlsanity.com/paper/2504.12285/) (2025)
- [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://mlsanity.com/paper/2406.03287/) (2024)
- [SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking](https://mlsanity.com/paper/2407.04752/) (2024)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [PowerQuant: Automorphism Search for Non-Uniform Quantization](https://mlsanity.com/paper/2301.09858/) (2023)
- [Spiking Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2306.17046/) (2023)
- [Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket](https://mlsanity.com/paper/2211.12933/) (2022)
- [Binarized Neural Machine Translation](https://mlsanity.com/paper/2302.04907/) (2023)
- [Q-S5: Towards Quantized State Space Models](https://mlsanity.com/paper/2406.09477/) (2024)
- [INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold](https://mlsanity.com/paper/2204.07439/) (2022)
- [NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search](https://mlsanity.com/paper/2308.05600/) (2023)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [REx: Data-Free Residual Quantization Error Expansion](https://mlsanity.com/paper/2203.14645/) (2022)
- [FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search](https://mlsanity.com/paper/2308.03290/) (2023)
- [Custom Gradient Estimators are Straight-Through Estimators in Disguise](https://mlsanity.com/paper/2405.05171/) (2024)
- [Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks](https://mlsanity.com/paper/2206.07741/) (2022)
- [Empirical Evaluation of Post-Training Quantization Methods for Language Tasks](https://mlsanity.com/paper/2210.16621/) (2022)
- [BitGNN: Unleashing the Performance Potential of Binary Graph Neural Networks on GPUs](https://mlsanity.com/paper/2305.02522/) (2023)
- [PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks](https://mlsanity.com/paper/2404.00103/) (2024)
- [Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping](https://mlsanity.com/paper/2308.07641/) (2023)
- [BOLD: Boolean Logic Deep Learning](https://mlsanity.com/paper/2405.16339/) (2024)
- [Spiking Diffusion Models](https://mlsanity.com/paper/2408.16467/) (2024)
- [Robust Training of Neural Networks at Arbitrary Precision and Sparsity](https://mlsanity.com/paper/2409.09245/) (2024)
- [Accelerating Error Correction Code Transformers](https://mlsanity.com/paper/2410.05911/) (2024)
- [Optimizing data-flow in Binary Neural Networks](https://mlsanity.com/paper/2304.00952/) (2023)
- [Boolean Variation and Boolean Logic BackPropagation](https://mlsanity.com/paper/2311.07427/) (2023)
- [PROM: Prioritize Reduction of Multiplications Over Lower Bit-Widths for Efficient CNNs](https://mlsanity.com/paper/2505.03254/) (2025)

## Abstract (arXiv)

Optimization of Top-1 ImageNet promotes enormous networks that may be impractical in inference settings. Binary neural networks (BNNs) have the potential to significantly lower the compute intensity but existing models suffer from low quality. To overcome this deficiency, we propose PokeConv, a binary convolution block which improves quality of BNNs by techniques such as adding multiple residual paths, and tuning the activation function. We apply it to ResNet-50 and optimize ResNet's initial convolutional layer which is hard to binarize. We name the resulting network family PokeBNN. These techniques are chosen to yield favorable improvements in both top-1 accuracy and the network's cost. In order to enable joint optimization of the cost together with accuracy, we define arithmetic computation effort (ACE), a hardware- and energy-inspired cost metric for quantized and binarized networks. We also identify a need to optimize an under-explored hyper-parameter controlling the binarization gradient approximation. We establish a new, strong state-of-the-art (SOTA) on top-1 accuracy together with commonly-used CPU64 cost, ACE cost and network size metrics. ReActNet-Adam, the previous SOTA in BNNs, achieved a 70.5% top-1 accuracy with 7.9 ACE. A small variant of PokeBNN achieves 70.5% top-1 with 2.6 ACE, more than 3x reduction in cost; a larger PokeBNN achieves 75.6% top-1 with 7.8 ACE, more than 5% improvement in accuracy without increasing the cost. PokeBNN implementation in JAX/Flax and reproduction instructions are available in AQT repository: https://github.com/google/aqt