mlsanityData to 14 Oct 2026

Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators

arXiv 2609.16085 ↗First posted 14 Sep 2026cs.AR cs.LG cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

Eight-bit integer (INT8) post-training quantization is the default recipe for edge deployment, under a widely held assumption: INT8 makes inference faster at a small, predictable accuracy cost, and a model quantized once can be carried to any target. We test that assumption with a controlled measurement study across seven hardware classes -- ARM and x86 CPUs, a discrete GPU, an NVIDIA Jetson AGX Orin iGPU and its NVDLA cores, and two vendor NPUs (Qualcomm Hexagon HTP, DEEPX DX-M1) -- holding the ONNX artifact and the quantization scales fixed so the integer kernel or ISA is the only free variable. Portability fails on three axes. (1) The sign of the INT8 speedup is set by the CPU's dot-product ISA (ARM dotprod/SDOT, x86 VNNI): cores that have it speed up by up to 2.1x, cores that lack it slow down by 1.7x, for the identical model and runtime. (2) INT8 outputs are not portable, and the rule is an invariance rather than a gradient: FP32 predictions are bit-identical for every pair (1000/1000), while INT8 predictions agree 1000/1000 exactly when two targets share an integer kernel and 958-965/1000 whenever they do not -- independent of whether the boundary is CPU<->CPU or CPU<->accelerator, and invisible to top-1 accuracy, which is preserved. (3) Vendor NPUs own quantization: a bring-your-own QDQ graph fails silently on one NPU (external scales ignored, accuracy 0.75 -> 0.005 while it compiles, profiles and runs without error) and loudly on the other (the compiler refuses the graph), so only the vendor's native path yields a correct engine. We further show that edge-NPU latency regimes are set by output/device-to-host transfer size rather than compute, and locate the transition with a fixed-compute sweep. We release the scripts and 32 reports. "Quantize once, deploy anywhere" is unsafe for embedded and automotive deployment, where per-input determinism and redundancy matter.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

28 of 43 references.

References · 28

  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  6. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  7. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  8. 2021A White Paper on Neural Network Quantizationcited by 472
  9. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  10. 2021Post-Training Quantization for Vision Transformercited by 278
  11. 2019MLPerf Inference Benchmarkcited by 199
  12. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
16 more
  1. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 179
  2. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  3. 2021MLPerf Tiny Benchmarkcited by 124
  4. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 77
  5. 2018AI Benchmark: Running Deep Neural Networks on Android Smartphonescited by 75
  6. 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 41
  7. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  8. 2018Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implicationscited by 36
  9. 2024Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applicationscited by 24
  10. 2024A Comprehensive Evaluation of Deep Learning Object Detection Models on Heterogeneous Edge Devicescited by 6
  11. 2019Integrating NVIDIA Deep Learning Accelerator (NVDLA) with RISC-V SoC on FireSimcited by 4
  12. 2025Quant-Trim in Practice: Improved Cross-Platform Low-Bit Deployment on Edge NPUscited by 2
  13. 2026What We Observe as LLM Behavior Can Be a Side-effect of Inference Backendcited by 2
  14. 2026The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inferencecited by 2
  15. 2025Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson -- Extendedcited by 1
  16. 2026Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformancecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023FP8 versus INT8 for efficient deep learning inference
  2. 2026Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0
  3. 2023SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference
  4. 2025Faster Inference of LLMs using FP8 on the Intel Gaudi
  5. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
  6. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
  7. 2026Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms
  8. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving