mlsanityData to 7 Oct 2026

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

arXiv 1606.06160 ↗First posted 20 Jun 2016cs.NE cs.LGcited by 557

Not on the map. Why?

Abstract (arXiv)

We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.

Key facts. Cited by 557 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 146; 2022: 117; 2023: 111; 2024: 76; 2025: 65; 2026: 42.

557 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  3. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  4. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  5. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  6. 2022FP8 Formats for Deep Learningcited by 187
  7. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  8. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  9. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  10. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  11. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  12. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
38 more
  1. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  2. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  3. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  4. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  5. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 87
  6. 2022FP8 Quantization: The Power of the Exponentcited by 87
  7. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  8. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  9. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  10. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  11. 2025Pretraining Large Language Models with NVFP4cited by 67
  12. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  13. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  14. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  15. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  16. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  17. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  18. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  19. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  20. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  21. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 57
  22. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  23. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 55
  24. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 53
  25. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  26. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 51
  27. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  28. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
  29. 2022Online Continual Learning for Embedded Devicescited by 48
  30. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  31. 2022FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videoscited by 46
  32. 2023Pruning vs Quantization: Which is Better?cited by 45
  33. 2021Zero-shot Adversarial Quantizationcited by 44
  34. 2021A comprehensive review of Binary Neural Networkcited by 44
  35. 2022Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Modelscited by 44
  36. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
  37. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  38. 2024Tiny Machine Learning: Progress and Futurescited by 40

50 most cited of 557 All 557 (JSON)