# DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos

arXiv: https://arxiv.org/abs/2203.03996 · HTML: https://mlsanity.com/paper/2203.03996/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [Fixed Point Quantization of Deep Convolutional Networks](https://mlsanity.com/paper/1511.06393/) (2015)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)
- [Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Video](https://mlsanity.com/paper/1807.06667/) (2018)
- [Low-Latency Video Semantic Segmentation](https://mlsanity.com/paper/1804.00389/) (2018)
- [Towards High Performance Video Object Detection](https://mlsanity.com/paper/1711.11577/) (2017)
- [Clockwork Convnets for Video Semantic Segmentation](https://mlsanity.com/paper/1608.03609/) (2016)
- [SBNet: Sparse Blocks Network for Fast Inference](https://mlsanity.com/paper/1801.02108/) (2018)
- [Skip-Convolutions for Efficient Video Processing](https://mlsanity.com/paper/2104.11487/) (2021)
- [Dynamic Kernel Distillation for Efficient Pose Estimation in Videos](https://mlsanity.com/paper/1908.09216/) (2019)
- [Minimum Energy Quantized Neural Networks](https://mlsanity.com/paper/1711.00215/) (2017)
- [Recurrent Residual Module for Fast Inference in Videos](https://mlsanity.com/paper/1802.09723/) (2018)
- [CBinfer: Exploiting Frame-to-Frame Locality for Faster Convolutional Network Inference on Video Streams](https://mlsanity.com/paper/1808.05488/) (2018)
- [Pack and Detect: Fast Object Detection in Videos Using Region-of-Interest Packing](https://mlsanity.com/paper/1809.01701/) (2018)

## Built on by (most cited first, 18 of 18)

- [Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models](https://mlsanity.com/paper/2211.02048/) (2022)
- [Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers](https://mlsanity.com/paper/2308.13494/) (2023)
- [MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos](https://mlsanity.com/paper/2210.09887/) (2022)
- [MaskVD: Region Masking for Efficient Video Object Detection](https://mlsanity.com/paper/2407.12067/) (2024)
- [D'ej`a Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reuse](https://mlsanity.com/paper/2506.14107/) (2025)
- [DeltaLLM: Delta-Key Sparse Attention for Edge LLMs](https://mlsanity.com/paper/2507.19608/) (2025)
- [Accelerated Event-Based Feature Detection and Compression for Surveillance Video Systems](https://mlsanity.com/paper/2312.08213/) (2023)
- [ResQ: Residual Quantization for Video Perception](https://mlsanity.com/paper/2308.09511/) (2023)
- [Sparse and Privacy-enhanced Representation for Human Pose Estimation](https://mlsanity.com/paper/2309.09515/) (2023)
- [Energy-Efficient & Real-Time Computer Vision with Intelligent Skipping via Reconfigurable CMOS Image Sensors](https://mlsanity.com/paper/2409.17341/) (2024)
- [EvConv: Fast CNN Inference on Event Camera Inputs For High-Speed Robot Perception](https://mlsanity.com/paper/2303.04670/) (2023)
- [Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs](https://mlsanity.com/paper/2307.14988/) (2023)
- [Uncertainty-Guided Spatial Pruning Architecture for Efficient Frame Interpolation](https://mlsanity.com/paper/2307.16555/) (2023)
- [Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation](https://mlsanity.com/paper/2312.07180/) (2023)
- [Learning K-U-Net with constant complexity: An Application to time series forecasting](https://mlsanity.com/paper/2410.02438/) (2024)
- [SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity](https://mlsanity.com/paper/2410.20790/) (2024)
- [ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering](https://mlsanity.com/paper/2506.13814/) (2025)
- [evMLP: An Efficient Event-Driven MLP Architecture for Vision](https://mlsanity.com/paper/2507.01927/) (2025)

## Abstract (arXiv)

Convolutional neural network inference on video data requires powerful hardware for real-time processing. Given the inherent coherence across consecutive frames, large parts of a video typically change little. By skipping identical image regions and truncating insignificant pixel updates, computational redundancy can in theory be reduced significantly. However, these theoretical savings have been difficult to translate into practice, as sparse updates hamper computational consistency and memory access coherence; which are key for efficiency on real hardware. With DeltaCNN, we present a sparse convolutional neural network framework that enables sparse frame-by-frame updates to accelerate video inference in practice. We provide sparse implementations for all typical CNN layers and propagate sparse feature updates end-to-end - without accumulating errors over time. DeltaCNN is applicable to all convolutional neural networks without retraining. To the best of our knowledge, we are the first to significantly outperform the dense reference, cuDNN, in practical settings, achieving speedups of up to 7x with only marginal differences in accuracy.