# Fast Algorithms for Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1509.09308 · HTML: https://mlsanity.com/paper/1509.09308/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2015.

## Built on by (most cited first, 30 of 102)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Scaling Local Self-Attention for Parameter Efficient Visual Backbones](https://mlsanity.com/paper/2103.12731/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations](https://mlsanity.com/paper/2304.11267/) (2023)
- [RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition](https://mlsanity.com/paper/2105.01883/) (2021)
- [AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning](https://mlsanity.com/paper/2107.08147/) (2021)
- [Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention](https://mlsanity.com/paper/2304.04237/) (2023)
- [Re-parameterizing Your Optimizers rather than Architectures](https://mlsanity.com/paper/2205.15242/) (2022)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality](https://mlsanity.com/paper/2112.11081/) (2021)
- [Tensor Program Optimization with Probabilistic Programs](https://mlsanity.com/paper/2205.13603/) (2022)
- [Dual-side Sparse Tensor Core](https://mlsanity.com/paper/2105.09564/) (2021)
- [BiViT: Extremely Compressed Binary Vision Transformer](https://mlsanity.com/paper/2211.07091/) (2022)
- [Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution](https://mlsanity.com/paper/2204.08397/) (2022)
- [Layer Collaboration in the Forward-Forward Algorithm](https://mlsanity.com/paper/2305.12393/) (2023)
- [A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators](https://mlsanity.com/paper/2105.12842/) (2021)
- [Zero-Knowledge Proof-based Verifiable Decentralized Machine Learning in Communication Network: A Comprehensive Survey](https://mlsanity.com/paper/2310.14848/) (2023)
- [1-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustness](https://mlsanity.com/paper/2311.16833/) (2023)
- [DiC: Rethinking Conv3x3 Designs in Diffusion Models](https://mlsanity.com/paper/2501.00603/) (2024)
- [Latent Space Explanation by Intervention](https://mlsanity.com/paper/2112.04895/) (2021)
- [GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity](https://mlsanity.com/paper/2108.11033/) (2021)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [Winograd Convolution for Deep Neural Networks: Efficient Point Selection](https://mlsanity.com/paper/2201.10369/) (2022)
- [Deformable Butterfly: A Highly Structured and Sparse Linear Transform](https://mlsanity.com/paper/2203.13556/) (2022)
- [Learning on Hardware: A Tutorial on Neural Network Accelerators and Co-Processors](https://mlsanity.com/paper/2104.09252/) (2021)
- [Winograd Convolution: A Perspective from Fault Tolerance](https://mlsanity.com/paper/2202.08675/) (2022)
- [Boosting DNN Cold Inference on Edge Devices](https://mlsanity.com/paper/2206.07446/) (2022)

## Abstract (arXiv)

Deep convolutional neural networks take GPU days of compute time to train on large data sets. Pedestrian detection for self driving cars requires very low latency. Image recognition for mobile phones is constrained by limited processing resources. The success of convolutional neural networks in these situations is limited by how fast we can compute them. Conventional FFT based convolution is fast for large filters, but state of the art convolutional neural networks use small, 3x3 filters. We introduce a new class of fast algorithms for convolutional neural networks using Winograd's minimal filtering algorithms. The algorithms compute minimal complexity convolution over small tiles, which makes them fast with small filters and small batch sizes. We benchmark a GPU implementation of our algorithm with the VGG network and show state of the art throughput at batch sizes from 1 to 64.