mlsanityData to 14 Oct 2026

Compacting Binary Neural Networks by Sparse Kernel Selection

arXiv 2303.14470 ↗First posted 25 Mar 2023cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Binary Neural Network (BNN) represents convolution weights with 1-bit values, which enhances the efficiency of storage and computation. This paper is motivated by a previously revealed phenomenon that the binary kernels in successful BNNs are nearly power-law distributed: their values are mostly clustered into a small number of codewords. This phenomenon encourages us to compact typical BNNs and obtain further close performance through learning non-repetitive kernels within a binary kernel subspace. Specifically, we regard the binarization process as kernel grouping in terms of a binary codebook, and our task lies in learning to select a smaller subset of codewords from the full codebook. We then leverage the Gumbel-Sinkhorn technique to approximate the codeword selection process, and develop the Permutation Straight-Through Estimator (PSTE) that is able to not only optimize the selection process end-to-end but also maintain the non-repetitive occupancy of selected codewords. Experiments verify that our method reduces both the model size and bit-wise computational costs, and achieves accuracy improvements compared with state-of-the-art BNNs under comparable budgets.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 25 Mar 2023.

Builds on

42 of 59 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  12. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
30 more
  1. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  2. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  3. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  4. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  5. 2018Slimmable Neural Networkscited by 305
  6. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  7. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  8. 2018Learning Latent Permutations with Gumbel-Sinkhorn Networkscited by 175
  9. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  10. 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
  11. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 112
  12. 2015Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networkscited by 80
  13. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 78
  14. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  15. 2020Rotated Binary Neural Networkcited by 77
  16. 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
  17. 2016Loss-aware Binarization of Deep Networkscited by 56
  18. 2020Searching for Low-Bit Weights in Quantized Neural Networkscited by 44
  19. 2018Bi-Real Net: Binarizing Deep Network Towards Real-Network Performancecited by 42
  20. 2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 39
  21. 2020Training Binary Neural Networks through Learning with Noisy Supervisioncited by 39
  22. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  23. 2020High-Capacity Expert Binary Networkscited by 34
  24. 2020BiDet: An Efficient Binarized Object Detectorcited by 33
  25. 2020Resolution Switchable Networks for Runtime Efficient Image Recognitioncited by 19
  26. 2021SiMaN: Sign-to-Magnitude Network Binarizationcited by 18
  27. 2019Searching for Accurate Binary Neural Architecturescited by 13
  28. 2020FleXOR: Trainable Fractional Quantizationcited by 9
  29. 2021Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networkscited by 8
  30. 2020Compressing Deep Convolutional Neural Networks by Stacking Low-dimensional Binary Convolution Filterscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  2. 2024ReActXGB: A Hybrid Binary Convolutional Neural Network Architecture for Improved Performance and Computational Efficiencycited by 0
  3. 2024Tiled Bit Networks: Sub-Bit Neural Network Compression Through Reuse of Learnable Binary Vectorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A comprehensive review of Binary Neural Network
  2. 2022Exploiting Kernel Compression on BNNs
  3. 2022Sparsifying Binary Networks
  4. 2023Binary domain generalization for sparsifying binary neural networks
  5. 2022Recurrent Bilinear Optimization for Binary Neural Networks
  6. 2023Optimizing data-flow in Binary Neural Networks
  7. 2022Multi-Objective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networks
  8. 2026Pruning Binarized Neural Networks: A Dedicated Framework and Globally Weighted Algorithms