mlsanityData to 7 Oct 2026

Neighborhood Attention Transformer

arXiv 2204.07143 ↗First posted 14 Apr 2022cs.CV cs.AI cs.LGcited by 235

Not on the map. Why?

Cited by 29 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision. NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels, and therefore enjoys a linear time and space complexity compared to the quadratic complexity of SA. The sliding-window pattern allows NA's receptive field to grow without needing extra pixel shifts, and preserves translational equivariance, unlike Swin Transformer's Window Self Attention (WSA). We develop NATTEN (Neighborhood Attention Extension), a Python package with efficient C++ and CUDA kernels, which allows NA to run up to 40% faster than Swin's WSA while using up to 25% less memory. We further present Neighborhood Attention Transformer (NAT), a new hierarchical transformer design based on NA that boosts image classification and downstream vision performance. Experimental results on NAT are competitive; NAT-Tiny reaches 83.2% top-1 accuracy on ImageNet, 51.4% mAP on MS-COCO and 48.4% mIoU on ADE20K, which is 1.9% ImageNet accuracy, 1.0% COCO mAP, and 2.6% ADE20K mIoU improvement over a Swin model with similar size. To support more research based on sliding-window attention, we open source our project and release our checkpoints at: https://github.com/SHI-Labs/Neighborhood-Attention-Transformer .

Key facts. Cited by 235 papers in the map (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

37 of 53 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2017Mask R-CNNcited by 7,259
25 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2017Deformable Convolutional Networkscited by 1,481
  11. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  12. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  13. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  14. 2017Random Erasing Data Augmentationcited by 1,235
  15. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  16. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  17. 2021Going deeper with Image Transformerscited by 759
  18. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  19. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  20. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  21. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  22. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  23. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  24. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  25. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 36; 2024: 60; 2025: 72; 2026: 50.

235 papers.

Sort:

By area:

  1. 2024VMamba: Visual State Space Modelcited by 955
  2. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  3. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  4. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  5. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  6. 2022MetaFormer Baselines for Visioncited by 131
  7. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 109
  8. 2023A Comprehensive Survey on Source-free Domain Adaptationcited by 104
  9. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  10. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  11. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  12. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
38 more
  1. 2022CiaoSR: Continuous Implicit Attention-in-Attention Network for Arbitrary-Scale Image Super-Resolutioncited by 69
  2. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  3. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 65
  4. 2022Dilated Neighborhood Attention Transformercited by 64
  5. 2024Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Modelcited by 62
  6. 2025Your ViT is Secretly an Image Segmentation Modelcited by 61
  7. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  8. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  9. 2024Context and Geometry Aware Voxel Transformer for Semantic Scene Completioncited by 50
  10. 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
  11. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  12. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  13. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  14. 2024Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusioncited by 40
  15. 2023Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attentioncited by 38
  16. 2023DilateFormer: Multi-Scale Dilated Transformer for Visual Recognitioncited by 34
  17. 2024Frequency-Adaptive Dilated Convolution for Semantic Segmentationcited by 34
  18. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  19. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
  20. 2024Breaking the Low-Rank Dilemma of Linear Attentioncited by 31
  21. 2024SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentationcited by 31
  22. 2023Matting Anythingcited by 30
  23. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 30
  24. 2024VSSD: Vision Mamba with Non-Causal State Space Dualitycited by 29
  25. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  26. 2023Semantic Image Segmentation: Two Decades of Researchcited by 26
  27. 2024CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Upcited by 26
  28. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  29. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  30. 2025OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernelscited by 24
  31. 2022Vicinity Vision Transformercited by 22
  32. 2025Block-Sparse Global Attention for Efficient Multi-View Geometry Transformerscited by 22
  33. 2022Prompt Tuning for Parameter-efficient Medical Image Segmentationcited by 21
  34. 2023DAT++: Spatially Dynamic Vision Transformer with Deformable Attentioncited by 21
  35. 2022VMFormer: End-to-End Video Matting with Transformercited by 20
  36. 2023Toward Real Text Manipulation Detection: New Dataset and New Solutioncited by 20
  37. 2023Point Cloud Classification Using Content-based Transformer via Clustering in Feature Spacecited by 19
  38. 2023Graph Transformer GANs for Graph-Constrained House Generationcited by 19

50 most cited of 235 All 235 (JSON)