# Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN

arXiv: https://arxiv.org/abs/2309.01439 · HTML: https://mlsanity.com/paper/2309.01439/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 80 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 30 of 69)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)

## Built on by (most cited first, 25 of 25)

- [The Tenth NTIRE 2025 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2504.10686/) (2025)
- [Underwater Image Restoration via Polymorphic Large Kernel CNNs](https://mlsanity.com/paper/2412.18459/) (2024)
- [Tensor Decomposition Based Attention Module for Spiking Neural Networks](https://mlsanity.com/paper/2310.14576/) (2023)
- [Optimizing Local-Global Dependencies for Accurate 3D Human Pose Estimation](https://mlsanity.com/paper/2412.19676/) (2024)
- [Low-light Stereo Image Enhancement and De-noising in the Low-frequency Information Enhanced Image Space](https://mlsanity.com/paper/2401.07753/) (2024)
- [Large coordinate kernel attention network for lightweight image super-resolution](https://mlsanity.com/paper/2405.09353/) (2024)
- [Real-time Stereo-based 3D Object Detection for Streaming Perception](https://mlsanity.com/paper/2410.12394/) (2024)
- [Moir'e Zero: An Efficient and High-Performance Neural Architecture for Moir'e Removal](https://mlsanity.com/paper/2507.22407/) (2025)
- [LKA-ReID:Vehicle Re-Identification with Large Kernel Attention](https://mlsanity.com/paper/2409.17908/) (2024)
- [BA-Net: Bridge Attention in Deep Neural Networks](https://mlsanity.com/paper/2410.07860/) (2024)
- [Learning Dynamic Local Context Representations for Infrared Small Target Detection](https://mlsanity.com/paper/2412.17401/) (2024)
- [Constrained Gaussian Process Motion Planning via Stein Variational Newton Inference](https://mlsanity.com/paper/2504.04936/) (2025)
- [LSP-ST: Ladder Shape-Biased Side-Tuning for Robust Infrared Small Target Detection](https://mlsanity.com/paper/2504.14481/) (2025)
- [Efficient Star Distillation Attention Network for Lightweight Image Super-Resolution](https://mlsanity.com/paper/2506.12475/) (2025)
- [BEVANet: Bilateral Efficient Visual Attention Network for Real-Time Semantic Segmentation](https://mlsanity.com/paper/2508.07300/) (2025)
- [MSMA: Multi-Scale Feature Fusion For Multi-Attribute 3D Face Reconstruction From Unconstrained Images](https://mlsanity.com/paper/2509.11763/) (2025)
- [FracDetNet: Advanced Fracture Detection via Dual-Focus Attention and Multi-scale Calibration in Medical X-ray Imaging](https://mlsanity.com/paper/2509.23416/) (2025)
- [Spatially-informed transformers: Injecting geostatistical covariance biases into self-attention for spatio-temporal forecasting](https://mlsanity.com/paper/2512.17696/) (2025)
- [UCAN: Unified Convolutional Attention Network for Expansive Receptive Fields in Lightweight Super-Resolution](https://mlsanity.com/paper/2603.11680/) (2026)
- [MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models](https://mlsanity.com/paper/2603.20074/) (2026)
- [Student Classroom Behavior Recognition Based on Improved YOLOv8s](https://mlsanity.com/paper/2604.27293/) (2026)
- [Multi-scale interaction network for stereo image super-resolution](https://mlsanity.com/paper/2605.21913/) (2026)
- [CF-YOLO: Context-Aware Feature Refinement for Camouflaged Industrial Micro-Defect Detection](https://mlsanity.com/paper/2608.28070/) (2026)
- [nnMNet: Baseline for Martian Terrain Semantic Segmentation](https://mlsanity.com/paper/2608.29609/) (2026)
- [HDMamba-YOLO: Efficient State-Space Perception and Local Spatial Reconstruction for UAV Small Object](https://mlsanity.com/paper/2609.23061/) (2026)

## Abstract (arXiv)

Visual Attention Networks (VAN) with Large Kernel Attention (LKA) modules have been shown to provide remarkable performance, that surpasses Vision Transformers (ViTs), on a range of vision-based tasks. However, the depth-wise convolutional layer in these LKA modules incurs a quadratic increase in the computational and memory footprints with increasing convolutional kernel size. To mitigate these problems and to enable the use of extremely large convolutional kernels in the attention modules of VAN, we propose a family of Large Separable Kernel Attention modules, termed LSKA. LSKA decomposes the 2D convolutional kernel of the depth-wise convolutional layer into cascaded horizontal and vertical 1-D kernels. In contrast to the standard LKA design, the proposed decomposition enables the direct use of the depth-wise convolutional layer with large kernels in the attention module, without requiring any extra blocks. We demonstrate that the proposed LSKA module in VAN can achieve comparable performance with the standard LKA module and incur lower computational complexity and memory footprints. We also find that the proposed LSKA design biases the VAN more toward the shape of the object than the texture with increasing kernel size. Additionally, we benchmark the robustness of the LKA and LSKA in VAN, ViTs, and the recent ConvNeXt on the five corrupted versions of the ImageNet dataset that are largely unexplored in the previous works. Our extensive experimental results show that the proposed LSKA module in VAN provides a significant reduction in computational complexity and memory footprints with increasing kernel size while outperforming ViTs, ConvNeXt, and providing similar performance compared to the LKA module in VAN on object recognition, object detection, semantic segmentation, and robustness tests.