mlsanityData to 14 Oct 2026

S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks

arXiv 2107.03453 ↗First posted 7 Jul 2021cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Shift neural networks reduce computation complexity by removing expensive multiplication operations and quantizing continuous weights into low-bit discrete values, which are fast and energy efficient compared to conventional neural networks. However, existing shift networks are sensitive to the weight initialization, and also yield a degraded performance caused by vanishing gradient and weight sign freezing problem. To address these issues, we propose S low-bit re-parameterization, a novel technique for training low-bit shift networks. Our method decomposes a discrete parameter in a sign-sparse-shift 3-fold manner. In this way, it efficiently learns a low-bit network with a weight dynamics similar to full-precision networks and insensitive to weight initialization. Our proposed training method pushes the boundaries of shift neural networks and shows 3-bit shift networks out-performs their full-precision counterparts in terms of top-1 accuracy on ImageNet.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 7 Jul 2021.

Builds on

28 of 43 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  5. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  6. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  7. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  8. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  9. 2016Trained Ternary Quantizationcited by 208
  10. 2016Ternary Weight Networkscited by 204
  11. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  12. 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
16 more
  1. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  2. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 117
  3. 2019AdderNet: Do We Really Need Multiplications in Deep Learning?cited by 82
  4. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  5. 2013Estimating or Propagating Gradients Through Stochastic Neuronscited by 55
  6. 2019DeepShift: Towards Multiplication-Less Neural Networkscited by 54
  7. 2020ShiftAddNet: A Hardware-Inspired Deep Networkcited by 52
  8. 2015Neural Networks with Few Multiplicationscited by 48
  9. 2020Searching for Low-Bit Weights in Quantized Neural Networkscited by 44
  10. 2017Learning Discrete Weights Using the Local Reparameterization Trickcited by 29
  11. 2018From Hashing to CNNs: Training BinaryWeight Networks via Hashingcited by 23
  12. 2017Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantizationcited by 21
  13. 2020Kernel Based Progressive Distillation for Adder Neural Networkscited by 19
  14. 2017ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networkscited by 13
  15. 2021AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligencecited by 10
  16. 2018Training Behavior of Sparse Neural Network Topologiescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 1; 2024: 0; 2025: 2; 2026: 1.

7 papers.

  1. 2022DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantizationcited by 5
  2. 2023On the Convergence of Stochastic Gradient Descent in Low-precision Number Formatscited by 4
  3. 2025PoTPTQ: A Two-step Power-of-Two Post-training for LLMscited by 3
  4. 2022Low-bit Shift Network for End-to-End Spoken Language Understandingcited by 1
  5. 2026PoTAcc: A Pipeline for End-to-End Acceleration of Power-of-Two Quantized DNNscited by 1
  6. 2022Standard Deviation-Based Quantization for Deep Neural Networkscited by 0
  7. 2025HaShiFlex: A High-Throughput Hardened Shifter DNN Accelerator with Fine-Tuning Flexibilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021SiMaN: Sign-to-Magnitude Network Binarization
  2. 2021Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks
  3. 2026Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
  4. 2022Boosting Binary Neural Networks via Dynamic Thresholds Learning
  5. 2024OvSW: Overcoming Silent Weights for Accurate Binary Neural Networks
  6. 2022Multi-Objective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networks
  7. 2023Post-Training Quantization for Re-parameterization via Coarse & Fine Weight Splitting
  8. 2024Designed Dithering Sign Activation for Binary Neural Networks