# Large-kernel Attention for Efficient and Robust Brain Lesion Segmentation

arXiv: https://arxiv.org/abs/2308.07251 · HTML: https://mlsanity.com/paper/2308.07251/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Visual Attention Network](https://mlsanity.com/paper/2202.09741/) (2022)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity](https://mlsanity.com/paper/2207.03620/) (2022)
- [Are Convolutional Neural Networks or Transformers more like human vision?](https://mlsanity.com/paper/2105.07197/) (2021)
- [ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset](https://mlsanity.com/paper/2206.06694/) (2022)
- [3D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2209.15076/) (2022)
- [Optimized U-Net for Brain Tumor Segmentation](https://mlsanity.com/paper/2110.03352/) (2021)
- [U-Net vs Transformer: Is U-Net Outdated in Medical Image Registration?](https://mlsanity.com/paper/2208.04939/) (2022)
- [On the Texture Bias for Few-Shot CNN Segmentation](https://mlsanity.com/paper/2003.04052/) (2020)
- [Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation](https://mlsanity.com/paper/2202.12295/) (2022)
- [Texture Bias Of CNNs Limits Few-Shot Classification Performance](https://mlsanity.com/paper/1910.08519/) (2019)

## Built on by (most cited first, 5 of 5)

- [Imbalance-Aware Culvert-Sewer Defect Segmentation Using an Enhanced Feature Pyramid Network](https://mlsanity.com/paper/2408.10181/) (2024)
- [Transformers-based architectures for stroke segmentation: A review](https://mlsanity.com/paper/2403.18637/) (2024)
- [Deep models for stroke segmentation: do complex architectures always perform better?](https://mlsanity.com/paper/2403.17177/) (2024)
- [Unified 3D MRI Representations via Sequence-Invariant Contrastive Learning](https://mlsanity.com/paper/2501.12057/) (2025)
- [Neuro-TransUNet: Segmentation of stroke lesion in MRI using transformers](https://mlsanity.com/paper/2406.06017/) (2024)

## Abstract (arXiv)

Vision transformers are effective deep learning models for vision tasks, including medical image segmentation. However, they lack efficiency and translational invariance, unlike convolutional neural networks (CNNs). To model long-range interactions in 3D brain lesion segmentation, we propose an all-convolutional transformer block variant of the U-Net architecture. We demonstrate that our model provides the greatest compromise in three factors: performance competitive with the state-of-the-art; parameter efficiency of a CNN; and the favourable inductive biases of a transformer. Our public implementation is available at https://github.com/liamchalcroft/MDUNet .