mlsanityData to 7 Oct 2026

Accelerating Deep Learning Model Inference on Arm CPUs with Ultra-Low Bit Quantization and Runtime

arXiv 2207.08820 ↗First posted 18 Jul 2022cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Deep Learning has been one of the most disruptive technological advancements in recent times. The high performance of deep learning models comes at the expense of high computational, storage and power requirements. Sensing the immediate need for accelerating and compressing these models to improve on-device performance, we introduce Deeplite Neutrino for production-ready optimization of the models and Deeplite Runtime for deployment of ultra-low bit quantized models on Arm-based platforms. We implement low-level quantization kernels for Armv7 and Armv8 architectures enabling deployment on the vast array of 32-bit and 64-bit Arm-based devices. With efficient implementations using vectorization, parallelization, and tiling, we realize speedups of up to 2x and 2.2x compared to TensorFlow Lite with XNNPACK backend on classification and detection models, respectively. We also achieve significant speedups of up to 5x and 3.2x compared to ONNX Runtime for classification and detection models, respectively.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 14 of its 32 references that are in the map. First posted on arXiv 18 Jul 2022.

Builds on

14 of 32 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  9. 2021A White Paper on Neural Network Quantizationcited by 453
  10. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  11. 2019Visual Wake Words Datasetcited by 46
  12. 2018Post-training 4-bit quantization of convolution networks for rapid-deploymentcited by 38
2 more
  1. 2021Do All MobileNets Quantize Poorly? Gaining Insights into the Effect of Quantization on Depthwise Separable Convolutional Networks Through the Eyes of Multi-scale Distributional Dynamicscited by 12
  2. 2021Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 1; 2026: 2.

6 papers.

  1. 2024Squat: Quant Small Language Models on the Edgecited by 16
  2. 2023DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tablescited by 8
  3. 2023Model Compression Methods for YOLOv5: A Reviewcited by 2
  4. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMVcited by 0
  5. 2026Does YOLO26 Truly Offer Advantages Over Its Predecessors for Edge Deployment? A Benchmark Study in Aquaculturecited by 0
  6. 2026Sustainable Edge Vision via Empirically Calibrated DVFS: Eliminating Thermal Throttling on Passively Cooled Hardwarecited by 0