mlsanityData to 7 Oct 2026

Loss-aware Binarization of Deep Networks

arXiv 1611.01600 ↗First posted 5 Nov 2016cs.NE cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time. Recently, there have been a number of attempts on binarizing the network weights and activations. This greatly reduces the network size, and replaces the underlying multiplications to additions or even XNOR bit operations. However, existing binarization schemes are based on simple matrix approximation and ignore the effect of binarization on the loss. In this paper, we propose a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss w.r.t. the binarized weights. The underlying proximal step has an efficient closed-form solution, and the second-order information can be efficiently obtained from the second moments already computed by the Adam optimizer. Experiments on both feedforward and recurrent networks show that the proposed loss-aware binarization algorithm outperforms existing binarization schemes, and is also more robust for wide and deep networks.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 15; 2023: 10; 2024: 6; 2025: 3; 2026: 3.

56 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  3. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  4. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  5. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  6. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  7. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  8. 2021A comprehensive review of Binary Neural Networkcited by 44
  9. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  10. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  11. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  12. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
38 more
  1. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  2. 2021Distribution-sensitive Information Retention for Accurate Binary Neural Networkcited by 25
  3. 2023BiBench: Benchmarking and Analyzing Network Binarizationcited by 22
  4. 2021Distance-aware Quantizationcited by 19
  5. 2023Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Trainingcited by 19
  6. 2021SiMaN: Sign-to-Magnitude Network Binarizationcited by 18
  7. 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
  8. 2023Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networkscited by 17
  9. 2022Adaptive Low-Precision Training for Embeddings in Click-Through Rate Predictioncited by 11
  10. 2022QReg: On Regularization Effects of Quantizationcited by 9
  11. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  12. 2021Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networkscited by 7
  13. 2021Training Structured Neural Networks Through Manifold Identification and Variance Reductioncited by 7
  14. 2022Mixed-Precision Neural Networks: A Surveycited by 7
  15. 2023Spatial-SpinDrop: Spatial Dropout-based Binary Bayesian Neural Network with Spintronics Implementationcited by 7
  16. 2021Compacting Deep Neural Networks for Internet of Things: Methods and Applicationscited by 6
  17. 2023Scale-Dropout: Estimating Uncertainty in Deep Neural Networks Using Stochastic Scalecited by 6
  18. 2023Learning Discretized Neural Networks under Ricci Flowcited by 4
  19. 2023Compacting Binary Neural Networks by Sparse Kernel Selectioncited by 3
  20. 2023Projected Stochastic Gradient Descent with Quantum Annealed Binary Gradientscited by 3
  21. 2024BOLD: Boolean Logic Deep Learningcited by 3
  22. 2021BCNN: Binary Complex Neural Networkcited by 2
  23. 2022FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representationcited by 2
  24. 2022PalQuant: Accelerating High-precision Networks on Low-precision Acceleratorscited by 2
  25. 2022PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networkscited by 2
  26. 2024One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Trainingcited by 2
  27. 2021Differentiable Architecture Pruning for Transfer Learningcited by 1
  28. 2022GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithmcited by 1
  29. 2023RedBit: An End-to-End Flexible Framework for Evaluating the Accuracy of Quantized CNNscited by 1
  30. 2024Quantized Approximately Orthogonal Recurrent Neural Networkscited by 1
  31. 2021Exact Backpropagation in Binary Weighted Networks with Group Weight Transformationscited by 0
  32. 2021Compact representations of convolutional neural networks via weight pruning and quantizationcited by 0
  33. 2022Searching Similarity Measure for Binarized Neural Networkscited by 0
  34. 2022A Closer Look at Hardware-Friendly Weight Quantizationcited by 0
  35. 2022LAB: Learnable Activation Binarizer for Binary Neural Networkscited by 0
  36. 2022Boosting Binary Neural Networks via Dynamic Thresholds Learningcited by 0
  37. 2023Understanding weight-magnitude hyperparameters in training binary networkscited by 0
  38. 2024CycleBNN: Cyclic Precision Training in Binary Neural Networkscited by 0

50 most cited of 56 All 56 (JSON)