# Highly Efficient 8-bit Low Precision Inference of Convolutional Neural Networks with IntelCaffe

arXiv: https://arxiv.org/abs/1805.08691 · HTML: https://mlsanity.com/paper/1805.08691/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2018.

## Built on by (most cited first, 5 of 5)

- [Neural Network Quantization for Efficient Inference: A Survey](https://mlsanity.com/paper/2112.06126/) (2021)
- [Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision](https://mlsanity.com/paper/2102.13565/) (2021)
- [How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Training](https://mlsanity.com/paper/2404.16898/) (2024)
- [RCT: Resource Constrained Training for Edge AI](https://mlsanity.com/paper/2103.14493/) (2021)
- [StruM: Structured Mixed Precision for Efficient Deep Learning Hardware Codesign](https://mlsanity.com/paper/2501.18953/) (2025)

## Abstract (arXiv)

High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents the efficient inference techniques of IntelCaffe, the first Intel optimized deep learning framework that supports efficient 8-bit low precision inference and model optimization techniques of convolutional neural networks on Intel Xeon Scalable Processors. The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining. We show that the inference throughput and latency with ResNet-50, Inception-v3 and SSD are improved by 1.38X-2.9X and 1.35X-3X respectively with neglectable accuracy loss from IntelCaffe FP32 baseline and by 56X-75X and 26X-37X from BVLC Caffe. All these techniques have been open-sourced on IntelCaffe GitHub1, and the artifact is provided to reproduce the result on Amazon AWS Cloud.