mlsanityData to 14 Oct 2026

Nearly Lossless Adaptive Bit Switching

arXiv 2502.01199 ↗First posted 3 Feb 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Model quantization is widely applied for compressing and accelerating deep neural networks (DNNs). However, conventional Quantization-Aware Training (QAT) focuses on training DNNs with uniform bit-width. The bit-width settings vary across different hardware and transmission demands, which induces considerable training and storage costs. Hence, the scheme of one-shot joint training multiple precisions is proposed to address this issue. Previous works either store a larger FP32 model to switch between different precision models for higher accuracy or store a smaller INT8 model but compromise accuracy due to using shared quantization parameters. In this paper, we introduce the Double Rounding quantization method, which fully utilizes the quantized representation range to accomplish nearly lossless bit-switching while reducing storage by using the highest integer precision instead of full precision. Furthermore, we observe a competitive interference among different precisions during one-shot joint training, primarily due to inconsistent gradients of quantization scales during backward propagation. To tackle this problem, we propose an Adaptive Learning Rate Scaling (ALRS) technique that dynamically adapts learning rates for various precisions to optimize the training process. Additionally, we extend our Double Rounding to one-shot mixed precision training and develop a Hessian-Aware Stochastic Bit-switching (HASB) strategy. Experimental results on the ImageNet-1K classification demonstrate that our methods have enough advantages to state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision. We also validate the feasibility of our method on detection and segmentation tasks, as well as on LLMs task. Our codes are available at https://github.com/haiduo/Double-Rounding.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

32 of 39 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Mask R-CNNcited by 7,277
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  5. 2019Learned Step Size Quantizationcited by 628
  6. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  7. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  8. 2017Large Batch Training of Convolutional Networkscited by 535
  9. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  10. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  11. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  12. 2018Slimmable Neural Networkscited by 305
20 more
  1. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  2. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  3. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 125
  4. 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 107
  5. 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 82
  6. 2019AdaBits: Neural Network Quantization with Adaptive Bit-Widthscited by 64
  7. 2020Robust Quantization: One Model to Rule Them Allcited by 63
  8. 2019Any-Precision Deep Neural Networkscited by 40
  9. 2021OMPQ: Orthogonal Mixed Precision Quantizationcited by 39
  10. 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26
  11. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  12. 2018A Quantization-Friendly Separable Convolution for MobileNetscited by 22
  13. 2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 22
  14. 2019Towards Efficient Training for Neural Network Quantizationcited by 16
  15. 2022Arbitrary Bit-width Network: A Joint Layer-Wise Quantization and Adaptive Inference Approachcited by 11
  16. 2023EQ-Net: Elastic Quantization Neural Networkscited by 10
  17. 2021Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widthscited by 5
  18. 2022Vertical Layering of Quantized Neural Networks for Heterogeneous Inferencecited by 4
  19. 2021One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustmentcited by 3
  20. 2020From Quantized DNNs to Quantizable DNNscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Compute-Optimal Quantization-Aware Training
  2. 2024AdaQAT: Adaptive Bit-Width Quantization-Aware Training
  3. 2023A Practical Mixed Precision Algorithm for Post-Training Quantization
  4. 2022AMED: Automatic Mixed-Precision Quantization for Edge Devices
  5. 2025Precision Neural Network Quantization via Learnable Adaptive Modules
  6. 2022Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks
  7. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
  8. 2024Retraining-free Model Quantization via One-Shot Weight-Coupling Learning