mlsanityData to 7 Oct 2026

Pareto-Optimal Quantized ResNet Is Mostly 4-bit

arXiv 2105.03536 ↗First posted 7 May 2021cs.LG cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

Quantization has become a popular technique to compress neural networks and reduce compute cost, but most prior work focuses on studying quantization without changing the network size. Many real-world applications of neural networks have compute cost and memory budgets, which can be traded off with model quality by changing the number of parameters. In this work, we use ResNet as a case study to systematically investigate the effects of quantization on inference compute cost-quality tradeoff curves. Our results suggest that for each bfloat16 ResNet model, there are quantized models with lower cost and higher accuracy; in other words, the bfloat16 compute cost-quality tradeoff curve is Pareto-dominated by the 4-bit and 8-bit curves, with models primarily quantized to 4-bit yielding the best Pareto curve. Furthermore, we achieve state-of-the-art results on ImageNet for 4-bit ResNet-50 with quantization-aware training, obtaining a top-1 eval accuracy of 77.09%. We demonstrate the regularizing effect of quantization by measuring the generalization gap. The quantization method we used is optimized for practicality: It requires little tuning and is designed with hardware capabilities in mind. Our work motivates further research into optimal numeric formats for quantization, as well as the development of machine learning accelerators supporting these formats. As part of this work, we contribute a quantization library written in JAX, which is open-sourced at https://github.com/google-research/google-research/tree/master/aqt.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

27 of 35 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  7. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  9. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  10. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  11. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  12. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
15 more
  1. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  2. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  3. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  4. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  5. 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 87
  6. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 76
  7. 2020Robust Quantization: One Model to Rule Them Allcited by 63
  8. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  9. 2018Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inferencecited by 33
  10. 2020FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activationscited by 30
  11. 2018NICE: Noise Injection and Clamping Estimation for Neural Network Quantizationcited by 24
  12. 2018Auto-tuning Neural Network Quantization Framework for Collaborative Inference Between the Cloud and Edgecited by 20
  13. 2021Confounding Tradeoffs for Neural Network Quantizationcited by 9
  14. 2021Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantizationcited by 6
  15. 2020Weight Equalizing Shift Scaler-Coupled Post-training Quantizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 5; 2024: 5; 2025: 7; 2026: 1.

26 papers.

Sort:

  1. 2022Efficiently Scaling Transformer Inferencecited by 529
  2. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  3. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
  4. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  5. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  6. 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 22
  7. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  8. 2025Matryoshka Quantizationcited by 11
  9. 2022QReg: On Regularization Effects of Quantizationcited by 9
  10. 2024Q-S5: Towards Quantized State Space Modelscited by 9
  11. 2022Online Model Compression for Federated Learning with Large Modelscited by 7
  12. 2025Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applicationscited by 6
14 more
  1. 2023FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Searchcited by 5
  2. 2025Task Vector Quantization for Memory-Efficient Model Mergingcited by 5
  3. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Sizecited by 4
  4. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formatscited by 4
  5. 2024PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networkscited by 3
  6. 2024Robust Training of Neural Networks at Arbitrary Precision and Sparsitycited by 3
  7. 2022Tabula: Efficiently Computing Nonlinear Activation Functions for Secure Neural Network Inferencecited by 2
  8. 2024Trainable Fixed-Point Quantization for Deep Learning Acceleration on FPGAscited by 1
  9. 2025Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantizationcited by 1
  10. 2025DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Schedulingcited by 1
  11. 2021Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compressioncited by 0
  12. 2023RAND: Robustness Aware Norm Decay For Quantized Seq2seq Modelscited by 0
  13. 2024Toward Efficient Convolutional Neural Networks With Structured Ternary Patternscited by 0
  14. 2026HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Modelscited by 0