# Dynamic Convolution: Attention over Convolution Kernels

arXiv: https://arxiv.org/abs/1912.03458 · HTML: https://mlsanity.com/paper/1912.03458/

## Key facts

- Cited by 356 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2019.

## Built on by (most cited first, 30 of 356)

- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Focal Modulation Networks](https://mlsanity.com/paper/2203.11926/) (2022)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers](https://mlsanity.com/paper/2205.03436/) (2022)
- [HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions](https://mlsanity.com/paper/2207.14284/) (2022)
- [Large Selective Kernel Network for Remote Sensing Object Detection](https://mlsanity.com/paper/2303.09030/) (2023)
- [Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution](https://mlsanity.com/paper/2203.14216/) (2022)
- [Lite-HRNet: A Lightweight High-Resolution Network](https://mlsanity.com/paper/2104.06403/) (2021)
- [End-to-End Vectorized HD-map Construction with Piecewise Bezier Curve](https://mlsanity.com/paper/2306.09700/) (2023)
- [Dynamic Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2202.09769/) (2022)
- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [Sparse Instance Activation for Real-Time Instance Segmentation](https://mlsanity.com/paper/2203.12827/) (2022)
- [Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation](https://mlsanity.com/paper/2307.09906/) (2023)
- [Towards Spatio-Temporal Aware Traffic Time Series Forecasting--Full Version](https://mlsanity.com/paper/2203.15737/) (2022)
- [UniHCP: A Unified Model for Human-Centric Perceptions](https://mlsanity.com/paper/2303.02936/) (2023)
- [Towards Stability of Autoregressive Neural Operators](https://mlsanity.com/paper/2306.10619/) (2023)
- [Omni-Dimensional Dynamic Convolution](https://mlsanity.com/paper/2209.07947/) (2022)
- [Rewrite the Stars](https://mlsanity.com/paper/2403.19967/) (2024)
- [Contrastive Grouping with Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2309.01017/) (2023)
- [Decoupled Dynamic Filter Networks](https://mlsanity.com/paper/2104.14107/) (2021)
- [Representing Volumetric Videos as Dynamic MLP Maps](https://mlsanity.com/paper/2304.06717/) (2023)
- [Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensus](https://mlsanity.com/paper/2207.01203/) (2022)
- [AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition](https://mlsanity.com/paper/2102.05775/) (2021)

## Abstract (arXiv)

Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability. To address this issue, we present Dynamic Convolution, a new design that increases model complexity without increasing the network depth or width. Instead of using a single convolution kernel per layer, dynamic convolution aggregates multiple parallel convolution kernels dynamically based upon their attentions, which are input dependent. Assembling multiple kernels is not only computationally efficient due to the small kernel size, but also has more representation power since these kernels are aggregated in a non-linear way via attention. By simply using dynamic convolution for the state-of-the-art architecture MobileNetV3-Small, the top-1 accuracy of ImageNet classification is boosted by 2.9% with only 4% additional FLOPs and 2.9 AP gain is achieved on COCO keypoint detection.