# Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2309.00121 · HTML: https://mlsanity.com/paper/2309.00121/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 63 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 30 of 46)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images](https://mlsanity.com/paper/1808.08114/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [Visual Attention Network](https://mlsanity.com/paper/2202.09741/) (2022)
- [Medical Image Segmentation Review: The success of U-Net](https://mlsanity.com/paper/2211.14830/) (2022)
- [DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation](https://mlsanity.com/paper/1506.06448/) (2015)
- [Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice](https://mlsanity.com/paper/2203.05962/) (2022)

## Built on by (most cited first, 30 of 30)

- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [MobileIE: An Extremely Lightweight and Effective ConvNet for Real-Time Image Enhancement on Mobile Devices](https://mlsanity.com/paper/2507.01838/) (2025)
- [D-Net: Dynamic Large Kernel with Dynamic Feature Fusion for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2403.10674/) (2024)
- [MSVM-UNet: Multi-Scale Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2408.13735/) (2024)
- [CENet: Context Enhancement Network for Medical Image Segmentation](https://mlsanity.com/paper/2505.18423/) (2025)
- [Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights](https://mlsanity.com/paper/2403.19882/) (2024)
- [LHU-Net: a Lean Hybrid U-Net for Cost-efficient, High-performance Volumetric Segmentation](https://mlsanity.com/paper/2404.05102/) (2024)
- [Shape-intensity knowledge distillation for robust medical image segmentation](https://mlsanity.com/paper/2409.17503/) (2024)
- [MetaSSC: Enhancing 3D Semantic Scene Completion for Autonomous Driving through Meta-Learning and Long-sequence Modeling](https://mlsanity.com/paper/2411.03672/) (2024)
- [Deep models for stroke segmentation: do complex architectures always perform better?](https://mlsanity.com/paper/2403.17177/) (2024)
- [Real Time Multi Organ Classification on Computed Tomography Images](https://mlsanity.com/paper/2404.18731/) (2024)
- [LKCell: Efficient Cell Nuclei Instance Segmentation with Large Convolution Kernels](https://mlsanity.com/paper/2407.18054/) (2024)
- [Spatial-Frequency Dual Progressive Attention Network For Medical Image Segmentation](https://mlsanity.com/paper/2406.07952/) (2024)
- [MLLA-UNet: Mamba-like Linear Attention in an Efficient U-Shape Model for Medical Image Segmentation](https://mlsanity.com/paper/2410.23738/) (2024)
- [CoRLD: Contrastive Representation Learning Of Deformable Shapes In Images](https://mlsanity.com/paper/2503.17162/) (2025)
- [NestedMorph: Enhancing Deformable Medical Image Registration with Nested Attention Mechanisms](https://mlsanity.com/paper/2410.02550/) (2024)
- [A4-Unet: Deformable Multi-Scale Attention Network for Brain Tumor Segmentation](https://mlsanity.com/paper/2412.06088/) (2024)
- [Nonparametric Teaching of Attention Learners](https://mlsanity.com/paper/2602.20461/) (2026)
- [Efficient 3D affinely equivariant CNNs with adaptive fusion of augmented spherical Fourier-Bessel bases](https://mlsanity.com/paper/2402.16825/) (2024)
- [Learning Dynamic Local Context Representations for Infrared Small Target Detection](https://mlsanity.com/paper/2412.17401/) (2024)
- [ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation](https://mlsanity.com/paper/2503.03327/) (2025)
- [Cross-Frequency Collaborative Training Network and Dataset for Semi-supervised First Molar Root Canal Segmentation](https://mlsanity.com/paper/2504.11856/) (2025)
- [ACM-UNet: Adaptive Integration of CNNs and Mamba for Efficient Medical Image Segmentation](https://mlsanity.com/paper/2505.24481/) (2025)
- [Efficient Feedback Gate Network for Hyperspectral Image Super-Resolution](https://mlsanity.com/paper/2506.17361/) (2025)
- [Airway Segmentation Network for Enhanced Tubular Feature Extraction](https://mlsanity.com/paper/2507.06581/) (2025)
- [MambaCAFU: Hybrid Multi-Scale and Multi-Attention Model with Mamba-Based Fusion for Medical Image Segmentation](https://mlsanity.com/paper/2510.03786/) (2025)
- [Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation](https://mlsanity.com/paper/2603.02727/) (2026)
- [Structure and Progress Aware Diffusion for Medical Image Segmentation](https://mlsanity.com/paper/2603.07889/) (2026)
- [MaLViL: Multi-axis Low-rank Vision-LSTM for Medical Image Segmentation](https://mlsanity.com/paper/2608.17635/) (2026)
- [TRIUNE-Net: Harmonizing Scale, Shape, and Efficiency in Pancreatic Tumor Segmentation](https://mlsanity.com/paper/2609.08303/) (2026)

## Abstract (arXiv)

Medical image segmentation has seen significant improvements with transformer models, which excel in grasping far-reaching contexts and global contextual information. However, the increasing computational demands of these models, proportional to the squared token count, limit their depth and resolution capabilities. Most current methods process D volumetric image data slice-by-slice (called pseudo 3D), missing crucial inter-slice information and thus reducing the model's overall performance. To address these challenges, we introduce the concept of Deformable Large Kernel Attention (D-LKA Attention), a streamlined attention mechanism employing large convolution kernels to fully appreciate volumetric context. This mechanism operates within a receptive field akin to self-attention while sidestepping the computational overhead. Additionally, our proposed attention mechanism benefits from deformable convolutions to flexibly warp the sampling grid, enabling the model to adapt appropriately to diverse data patterns. We designed both 2D and 3D adaptations of the D-LKA Attention, with the latter excelling in cross-depth data understanding. Together, these components shape our novel hierarchical Vision Transformer architecture, the textitD-LKA Net. Evaluations of our model against leading methods on popular medical segmentation datasets (Synapse, NIH Pancreas, and Skin lesion) demonstrate its superior performance. Our code implementation is publicly available at the: https://github.com/mindflow-institue/deformableLKA