# Accelerating Deep Learning Inference via Freezing

arXiv: https://arxiv.org/abs/2002.02645 · HTML: https://mlsanity.com/paper/2002.02645/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2020.

## Built on by (most cited first, 10 of 10)

- [Dynamic Sparse Training with Structured Sparsity](https://mlsanity.com/paper/2305.02299/) (2023)
- [Layer-wise Adaptive Model Aggregation for Scalable Federated Learning](https://mlsanity.com/paper/2110.10302/) (2021)
- [GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge](https://mlsanity.com/paper/2201.07705/) (2022)
- [ECG-Image-Kit: A Synthetic Image Generation Toolbox to Facilitate Deep Learning-Based Electrocardiogram Digitization](https://mlsanity.com/paper/2307.01946/) (2023)
- [Accelerating Deep Learning Inference via Learned Caches](https://mlsanity.com/paper/2101.07344/) (2021)
- [MadEye: Boosting Live Video Analytics Accuracy with Adaptive Camera Configurations](https://mlsanity.com/paper/2304.02101/) (2023)
- [Embedding Recycling for Language Models](https://mlsanity.com/paper/2207.04993/) (2022)
- [Approximate Caching for Efficiently Serving Diffusion Models](https://mlsanity.com/paper/2312.04429/) (2023)
- [Accelerating Deep Learning Classification with Error-controlled Approximate-key Caching](https://mlsanity.com/paper/2112.06671/) (2021)
- [Improving the Performance of DNN-based Software Services using Automated Layer Caching](https://mlsanity.com/paper/2209.08625/) (2022)

## Abstract (arXiv)

Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous owing to their high accuracy on real-world tasks. However, this increase in accuracy comes at the cost of computationally expensive models leading to higher prediction latencies. Prior efforts to reduce this latency such as quantization, model distillation, and any-time prediction models typically trade-off accuracy for performance. In this work, we observe that caching intermediate layer outputs can help us avoid running all the layers of a DNN for a sizeable fraction of inference requests. We find that this can potentially reduce the number of effective layers by half for 91.58% of CIFAR-10 requests run on ResNet-18. We present Freeze Inference, a system that introduces approximate caching at each intermediate layer and we discuss techniques to reduce the cache size and improve the cache hit rate. Finally, we discuss some of the open research challenges in realizing such a design.