# SMM-Conv: Scalar Matrix Multiplication with Zero Packing for Accelerated Convolution

arXiv: https://arxiv.org/abs/2411.15659 · HTML: https://mlsanity.com/paper/2411.15659/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [Fast Algorithms for Convolutional Neural Networks](https://mlsanity.com/paper/1509.09308/) (2015)
- [YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers](https://mlsanity.com/paper/1811.05588/) (2018)
- [MEC: Memory-efficient Convolution for Deep Neural Network](https://mlsanity.com/paper/1706.06873/) (2017)
- [High Performance Zero-Memory Overhead Direct Convolutions](https://mlsanity.com/paper/1809.10170/) (2018)
- [Hypernetwork-Based Adaptive Image Restoration](https://mlsanity.com/paper/2206.05970/) (2022)
- [The Role of Redundant Bases and Shrinkage Functions in Image Denoising](https://mlsanity.com/paper/2004.10306/) (2020)

## Abstract (arXiv)

We present a novel approach for accelerating convolutions during inference for CPU-based architectures. The most common method of computation involves packing the image into the columns of a matrix (im2col) and performing general matrix multiplication (GEMM) with a matrix of weights. This results in two main drawbacks: (a) im2col requires a large memory buffer and can experience inefficient memory access, and (b) while GEMM is highly optimized for scientific matrices multiplications, it is not well suited for convolutions. We propose an approach that takes advantage of scalar-matrix multiplication and reduces memory overhead. Our experiments with commonly used network architectures demonstrate a significant speedup compared to existing indirect methods.