QuantTune: Optimizing Model Quantization with Adaptive Outlier-Driven Fine Tuning
Not on the map. Why?
Abstract (arXiv)
Transformer-based models have gained widespread popularity in both the computer vision (CV) and natural language processing (NLP) fields. However, significant challenges arise during post-training linear quantization, leading to noticeable reductions in inference accuracy. Our study focuses on uncovering the underlying causes of these accuracy drops and proposing a quantization-friendly fine-tuning method, QuantTune. Firstly, our analysis revealed that, on average, 65% of quantization errors result from the precision loss incurred by the dynamic range amplification effect of outliers across the target Transformer-based models. Secondly, QuantTune adjusts weights based on the deviation of outlier activations and effectively constrains the dynamic ranges of the problematic activations. As a result, it successfully mitigates the negative impact of outliers on the inference accuracy of quantized models. Lastly, QuantTune can be seamlessly integrated into the back-propagation pass in the fine-tuning process without requiring extra complexity in inference software and hardware design. Our approach showcases significant improvements in post-training quantization across a range of Transformer-based models, including ViT, Bert-base, and OPT. QuantTune reduces accuracy drops by 12.09% at 8-bit quantization and 33.8% at 7-bit compared to top calibration methods, outperforming state-of-the-art solutions by over 18.84% across ViT models.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 53 of its 64 references that are in the map. First posted on arXiv 11 Mar 2024.
Builds on
53 of 64 references.
References · 53
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
38 more
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
- 2019Learned Step Size Quantizationcited by 604
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
- 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
- 2021A White Paper on Neural Network Quantizationcited by 453
- 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
- 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
- 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
- 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
- 2021Post-Training Quantization for Vision Transformercited by 274
- 2023A Survey on Model Compression for Large Language Modelscited by 263
- 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
- 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
- 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
- 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
- 2022Compute Trends Across Three Eras of Machine Learningcited by 161
- 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
- 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
- 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
- 2021Patch Slimming for Efficient Vision Transformerscited by 131
- 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
- 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
- 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
- 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
- 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
- 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
- 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
- 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
- 2020Robust Quantization: One Model to Rule Them Allcited by 63
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.