mlsanityData to 14 Oct 2026

FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs

arXiv 2211.06982 ↗First posted 13 Nov 2022cs.PF cs.LG cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Although prior art has demonstrated negligible accuracy drop in sub-byte quantization -- where weights and/or activations are represented by less than 8 bits -- popular SIMD instructions of CPUs do not natively support these datatypes. While recent methods, such as ULPPACK, are already using sub-byte quantization on general-purpose CPUs with vector units, they leave out several empty bits between the sub-byte values in memory and in vector registers to avoid overflow to the neighbours during the operations. This results in memory footprint and bandwidth-usage inefficiencies and suboptimal performance. In this paper, we present memory layouts for storing, and mechanisms for processing sub-byte (4-, 2-, or 1-bit) models that utilize all the bits in the memory as well as in the vector registers for the actual data. We provide compute kernels for the proposed layout for the GEMV (GEneral Matrix-Vector multiplication) operations between weights and activations of different datatypes (e.g., 8-bit activations and 4-bit weights). For evaluation, we extended the TFLite package and added our methods to it, then ran the models on the cycle-accurate gem5 simulator to compare detailed memory and CPU cycles of each method. We compare against nine other methods that are actively used in production including GEMLOWP, Ruy, XNNPack, and ULPPACK. Furthermore, we explore the effect of different input and output sizes of deep learning layers on the performance of our proposed method. Experimental results show 0.96-2.1x speedup for small sizes and 1.2-6.7x speedup for mid to large sizes. Applying our proposal to a real-world speech recognition model, Mozilla DeepSpeech, we proved that our method achieves 1.56-2.11x end-to-end speedup compared to the state-of-the-art, depending on the bit-width employed.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 13 Nov 2022.

Builds on

16 of 32 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  4. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  5. 2019Learned Step Size Quantizationcited by 628
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  7. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  8. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  9. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  10. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 160
  11. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  12. 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 41
4 more
  1. 2020Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Frameworkcited by 27
  2. 2020BitPruning: Learning Bitlengths for Aggressive and Accurate Quantizationcited by 14
  3. 2020The gem5 Simulator: Version 20.0+cited by 14
  4. 2018Effective Quantization Approaches for Recurrent Neural Networkscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
  2. 2023Sparq: A Custom RISC-V Vector Processor for Efficient Sub-Byte Quantized Inference
  3. 2026RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication
  4. 2023Quark: An Integer RISC-V Vector Processor for Sub-Byte Quantized DNN Inference
  5. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
  6. 2025The Cambrian Explosion of Mixed-Precision Matrix Multiplication for Quantized Deep Learning Inference
  7. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
  8. 2025VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference