mlsanityData to 7 Oct 2026

FIT: A Metric for Model Sensitivity

arXiv 2210.08502 ↗First posted 16 Oct 2022cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Model compression is vital to the deployment of deep learning on edge devices. Low precision representations, achieved via quantization of weights and activations, can reduce inference time and memory requirements. However, quantifying and predicting the response of a model to the changes associated with this procedure remains challenging. This response is non-linear and heterogeneous throughout the network. Understanding which groups of parameters and activations are more sensitive to quantization than others is a critical stage in maximizing efficiency. For this purpose, we propose FIT. Motivated by an information geometric perspective, FIT combines the Fisher information with a model of quantization. We find that FIT can estimate the final performance of a network without retraining. FIT effectively fuses contributions from both parameter and activation quantization into a single metric. Additionally, FIT is fast to compute when compared to existing methods, demonstrating favourable convergence properties. These properties are validated experimentally across hundreds of quantization configurations, with a focus on layer-wise mixed-precision quantization.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 16 Oct 2022.

Builds on

28 of 43 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  8. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  9. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  10. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  11. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  12. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
16 more
  1. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
  2. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  3. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  4. 2020Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectorscited by 78
  5. 2018An elementary introduction to information geometrycited by 76
  6. 2018AI Benchmark: Running Deep Neural Networks on Android Smartphonescited by 75
  7. 2019AutoQ: Automated Kernel-Wise Neural Network Quantizationcited by 60
  8. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 53
  9. 2016Towards the Limit of Network Quantizationcited by 49
  10. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 43
  11. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  12. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  13. 2021BN-NAS: Neural Architecture Search with Batch Normalizationcited by 21
  14. 2016Reducing the Model Order of Deep Neural Networks Using Information Theorycited by 10
  15. 2021BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratchcited by 10
  16. 2020Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 2; 2026: 1.

10 papers.

  1. 2024Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labelingcited by 24
  2. 2023A Practical Mixed Precision Algorithm for Post-Training Quantizationcited by 23
  3. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inferencecited by 9
  4. 2023QBitOpt: Fast and Accurate Bitwidth Reallocation during Trainingcited by 4
  5. 2023Towards Optimal Compression: Joint Pruning and Quantizationcited by 3
  6. 2023Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantizationcited by 2
  7. 2025RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Modelscited by 2
  8. 2024CRoP: Context-wise Robust Static Human-Sensing Personalizationcited by 1
  9. 2024Weight Block Sparsity: Training, Compilation, and AI Engine Acceleratorscited by 0
  10. 2026SDFP: Speculative Decoding with FIT-Pruned Models for Training-Free and Plug-and-Play LLM Accelerationcited by 0