# Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Serving

arXiv: https://arxiv.org/abs/2311.18174 · HTML: https://mlsanity.com/paper/2311.18174/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 50 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [TensorFlow-Serving: Flexible, High-Performance ML Serving](https://mlsanity.com/paper/1712.06139/) (2017)
- [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://mlsanity.com/paper/1802.04730/) (2018)
- [Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More](https://mlsanity.com/paper/2103.10891/) (2021)
- [Exploiting Parallelism Opportunities with Deep Learning Frameworks](https://mlsanity.com/paper/1908.04705/) (2019)
- [Optimizing Inference Performance of Transformers on CPUs](https://mlsanity.com/paper/2102.06621/) (2021)
- [Auto-tuning TensorFlow Threading Model for CPU Backend](https://mlsanity.com/paper/1812.01665/) (2018)
- [Energy Efficiency Features of the Intel Skylake-SP Processor and Their Impact on Performance](https://mlsanity.com/paper/1905.12468/) (2019)

## Built on by (most cited first, 2 of 2)

- [D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving](https://mlsanity.com/paper/2403.01876/) (2024)
- [SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services](https://mlsanity.com/paper/2501.02181/) (2025)

## Abstract (arXiv)

In this paper, we investigate how to push the performance limits of serving Deep Neural Network (DNN) models on CPU-based servers. Specifically, we observe that while intra-operator parallelism across multiple threads is an effective way to reduce inference latency, it provides diminishing returns. Our primary insight is that instead of running a single instance of a model with all available threads on a server, running multiple instances each with smaller batch sizes and fewer threads for intra-op parallelism can provide lower inference latency. However, the right configuration is hard to determine manually since it is workload- (DNN model and batch size used by the serving system) and deployment-dependent (number of CPU cores on server). We present Packrat, a new serving system for online inference that given a model and batch size (B) algorithmically picks the optimal number of instances (i), the number of threads each should be allocated (t), and the batch sizes each should operate on (b) that minimizes latency. Packrat is built as an extension to TorchServe and supports online reconfigurations to avoid serving downtime. Averaged across a range of batch sizes, Packrat improves inference latency by 1.43× to 1.83× on a range of commonly used DNNs.