Automatic mixed precision for optimizing gained time with constrained loss mean-squared-error based on model partition to sequential sub-graphs
Not on the map. Why?
Abstract (arXiv)
Quantization is essential for Neural Network (NN) compression, reducing model size and computational demands by using lower bit-width data types, though aggressive reduction often hampers accuracy. Mixed Precision (MP) mitigates this tradeoff by varying the numerical precision across network layers. This study focuses on automatically selecting an optimal MP configuration within Post-Training Quantization (PTQ) for inference. The first key contribution is a novel sensitivity metric derived from a first-order Taylor series expansion of the loss function as a function of quantization errors in weights and activations. This metric, based on the Mean Square Error (MSE) of the loss, is efficiently calculated per layer using high-precision forward and backward passes over a small calibration dataset. The metric is additive across layers, with low calibration memory overhead as weight optimization is unnecessary. The second contribution is an accurate hardware-aware method for predicting MP time gain by modeling it as additive for sequential sub-graphs. An algorithm partitions the model graph into sequential subgraphs, measuring time gain for each configuration using a few samples. After calibrating per-layer sensitivity and time gain, an Integer Programming (IP) problem is formulated to maximize time gain while keeping loss MSE below a set threshold. Memory gain and theoretical time gain based on Multiply and Accumulate (MAC) operations are also considered. Rigorous experiments on the Intel Gaudi 2 accelerator validate the approach on several Large Language Models (LLMs).
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 21 references that are in the map. First posted on arXiv 19 May 2025.
Builds on
13 of 21 references.
References · 13
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
- 2021A White Paper on Neural Network Quantizationcited by 472
- 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
- 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
- 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 61
- 2023A Practical Mixed Precision Algorithm for Post-Training Quantizationcited by 24
- 2021Neural Network Quantization for Efficient Inference: A Surveycited by 14
- 2024EfQAT: An Efficient Framework for Quantization-Aware Trainingcited by 7
- 2025Faster Inference of LLMs using FP8 on the Intel Gaudicited by 2
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Mixed Precision Post Training Quantization of Neural Networks with Sensitivity Guided Search
- 2021Data-free mixed-precision quantization using novel sensitivity metric
- 2022Mixed-Precision Inference Quantization: Radically Towards Faster inference speed, Lower Storage requirement, and Lower Loss
- 2022AMED: Automatic Mixed-Precision Quantization for Edge Devices
- 2025Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models
- 2023Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantization
- 2023Efficient and Effective Methods for Mixed Precision Neural Network Quantization for Faster, Energy-efficient Inference
- 2024Retraining-free Model Quantization via One-Shot Weight-Coupling Learning