mlsanityData to 7 Oct 2026

Regularizing Activation Distribution for Training Binarized Deep Networks

arXiv 1904.02823 ↗First posted 4 Apr 2019cs.CVcited by 61

Not on the map. Why?

Abstract (arXiv)

Binarized Neural Networks (BNNs) can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN's energy efficiency. In this paper, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 14; 2023: 16; 2024: 9; 2025: 3; 2026: 2.

61 papers.

Sort:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 51
  3. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  4. 2021A comprehensive review of Binary Neural Networkcited by 44
  5. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  6. 2021Distribution-sensitive Information Retention for Accurate Binary Neural Networkcited by 25
  7. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  8. 2023Binarized Spectral Compressive Imagingcited by 23
  9. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  10. 2022EcoFormer: Energy-Saving Attention with Linear Complexitycited by 21
  11. 2021Information Theoretic Representation Distillationcited by 19
  12. 2023Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Trainingcited by 19
38 more
  1. 2021SiMaN: Sign-to-Magnitude Network Binarizationcited by 18
  2. 2024Lightweight Pixel Difference Networks for Efficient Visual Representation Learningcited by 17
  3. 2021S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibrationcited by 15
  4. 2022Lipschitz Continuity Retained Binary Neural Networkcited by 14
  5. 2022Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticketcited by 13
  6. 2022Network Binarization via Contrastive Learningcited by 12
  7. 2024BiPer: Binary Neural Networks using a Periodic Functioncited by 9
  8. 2021Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networkscited by 7
  9. 2023Spatial-SpinDrop: Spatial Dropout-based Binary Bayesian Neural Network with Spintronics Implementationcited by 7
  10. 2021Single-path Bit Sharing for Automatic Loss-aware Model Compressioncited by 6
  11. 2021EZCrop: Energy-Zoned Channels for Robust Output Pruningcited by 6
  12. 2023Scale-Dropout: Estimating Uncertainty in Deep Neural Networks Using Stochastic Scalecited by 6
  13. 2024Custom Gradient Estimators are Straight-Through Estimators in Disguisecited by 6
  14. 2022BiFSMN: Binary Neural Network for Keyword Spottingcited by 5
  15. 2023Stochastic Configuration Machines for Industrial Artificial Intelligencecited by 5
  16. 2024A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Networkcited by 5
  17. 2021Self-Distribution Binary Neural Networkscited by 4
  18. 2022BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performancecited by 4
  19. 2021Improving Binary Neural Networks through Fully Utilizing Latent Weightscited by 3
  20. 2021BNAS v2: Learning Architectures for Binary Networks with Empirical Improvementscited by 3
  21. 2023Compacting Binary Neural Networks by Sparse Kernel Selectioncited by 3
  22. 2024BOLD: Boolean Logic Deep Learningcited by 3
  23. 2021Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizercited by 2
  24. 2022BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformationcited by 2
  25. 2022AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networkscited by 2
  26. 2023Fixed-point quantization aware training for on-device keyword-spottingcited by 2
  27. 2023Optimizing data-flow in Binary Neural Networkscited by 2
  28. 2023AdaFlood: Adaptive Flood Regularizationcited by 2
  29. 2024OvSW: Overcoming Silent Weights for Accurate Binary Neural Networkscited by 2
  30. 2021Width Transfer: On the (In)variance of Width Optimizationcited by 1
  31. 2023RedBit: An End-to-End Flexible Framework for Evaluating the Accuracy of Quantized CNNscited by 1
  32. 2023Training Neural Networks with Universal Adiabatic Quantum Computingcited by 1
  33. 2023BinaryHPE: 3D Human Pose and Shape Estimation via Binarizationcited by 1
  34. 2025BiVM: Accurate Binarized Neural Network for Efficient Video Mattingcited by 1
  35. 2022Combinatorial optimization for low bit-width neural networkscited by 0
  36. 2022Sparsifying Binary Networkscited by 0
  37. 2022IR2Net: Information Restriction and Information Recovery for Accurate Binary Neural Networkscited by 0
  38. 2022LAB: Learnable Activation Binarizer for Binary Neural Networkscited by 0

50 most cited of 61 All 61 (JSON)