mlsanityData to 7 Oct 2026

Vertical Layering of Quantized Neural Networks for Heterogeneous Inference

arXiv 2212.05326 ↗First posted 10 Dec 2022cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Although considerable progress has been obtained in neural network quantization for efficient inference, existing methods are not scalable to heterogeneous devices as one dedicated model needs to be trained, transmitted, and stored for one specific hardware setting, incurring considerable costs in model training and maintenance. In this paper, we study a new vertical-layered representation of neural network weights for encapsulating all quantized models into a single one. With this representation, we can theoretically achieve any precision network for on-demand service while only needing to train and maintain one model. To this end, we propose a simple once quantization-aware training (QAT) scheme for obtaining high-performance vertical-layered models. Our design incorporates a cascade downsampling mechanism which allows us to obtain multiple quantized networks from one full precision source model by progressively mapping the higher precision weights to their adjacent lower precision counterparts. Then, with networks of different bit-widths from one source model, multi-objective optimization is employed to train the shared source model weights such that they can be updated simultaneously, considering the performance of all networks. By doing this, the shared weights will be optimized to balance the performance of different quantized models, thus making the weights transferable among different bit widths. Experiments show that the proposed vertical-layered representation and developed once QAT scheme are effective in embodying multiple quantized networks into a single one and allow one-time training, and it delivers comparable performance as that of quantized models tailored to any specific bit-width. Code will be available.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 10 Dec 2022.

Builds on

54 of 65 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2016Wide Residual Networkscited by 3,057
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  11. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
38 more
  1. 2019Learned Step Size Quantizationcited by 604
  2. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  3. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  4. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  5. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  6. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  7. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  8. 2021Dynamic Neural Networks: A Surveycited by 405
  9. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  10. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  11. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  12. 2018Slimmable Neural Networkscited by 301
  13. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  14. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  15. 2016Trained Ternary Quantizationcited by 203
  16. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  17. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  18. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  19. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  20. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  21. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  22. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
  23. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 98
  24. 2019Loss Aware Post-training Quantizationcited by 94
  25. 2018Channel Gating Neural Networkscited by 80
  26. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  27. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  28. 2019AdaBits: Neural Network Quantization with Adaptive Bit-Widthscited by 63
  29. 2020HAWQV3: Dyadic Neural Network Quantizationcited by 59
  30. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 57
  31. 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 49
  32. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  33. 2018Bi-Real Net: Binarizing Deep Network Towards Real-Network Performancecited by 42
  34. 2019QKD: Quantization-aware Knowledge Distillationcited by 42
  35. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
  36. 2019Fighting Quantization Bias With Biascited by 30
  37. 2019Improving One-shot NAS by Suppressing the Posterior Fadingcited by 28
  38. 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Viewscited by 7
  2. 2025Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detectioncited by 5
  3. 2025Nearly Lossless Adaptive Bit Switchingcited by 0
  4. 2025Hierarchical Spatial Algorithms for High-Resolution Image Quantization and Feature Extractioncited by 0