# A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

arXiv: https://arxiv.org/abs/2608.26926 · HTML: https://mlsanity.com/paper/2608.26926/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 17 of 17)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://mlsanity.com/paper/2407.11062/) (2024)
- [Compressing Large-Scale Transformer-Based Models: A Case Study on BERT](https://mlsanity.com/paper/2002.11985/) (2020)
- [HAWQV3: Dyadic Neural Network Quantization](https://mlsanity.com/paper/2011.10680/) (2020)
- [Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques](https://mlsanity.com/paper/2411.06084/) (2024)
- [CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression](https://mlsanity.com/paper/2510.12721/) (2025)
- [GEO-Flag: Detecting and Measuring GEO-Optimized Web Content](https://mlsanity.com/paper/2608.16824/) (2026)
- [HAPM -- Hardware Aware Pruning Method for CNN hardware accelerators in resource constrained devices](https://mlsanity.com/paper/2408.14055/) (2024)

## Built on by (most cited first, 1 of 1)

- [A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint](https://mlsanity.com/paper/2608.28003/) (2026)

## Abstract (arXiv)

Small language models (sLLMs) are nowadays hosted on devices with limited memory and computational budget. In an autoregressive setup, inference is memory-bandwidth bound: uniform quantization is often detrimental to such models, since their architecture has limited redundancies and only a few layers are not very sensitive to lower precision. We propose a composite metric that combines two orthogonal criteria: information retention (measured in terms of a normalized SQNR-based coefficient) and throughput gains (modeled using a roofline-based latency analysis). By profiling Gemma 3 1B, we find that Feed-Forward Network blocks and the embedding matrix are the most promising targets for acceleration. For each candidate, we estimate a normalized quality score based on simulated quantization and a normalized speed score based on roofline modeling with no actual execution needed. We combine the two scores in a composite priority coefficient, allowing us to tune the trade-off between speed and quality as needed. Our metric is general and can be used to prioritize individual blocks, their projection sublayers, or transformer layers as a whole. We evaluate our approach on several model architectures, showing that our estimates have at around 4% prediction error for the accelerated speedup. We find that our method generally allocates more resources to the most expressive layers compared to evolutionary search, specialized accelerators, or Shapley-value-based approaches that require expensive approximate inference. Our analytical approach makes sLLM quantization a predictable engineering task.