mlsanityData to 7 Oct 2026

Window Attention is Bugged: How not to Interpolate Position Embeddings

arXiv 2311.05613 ↗First posted 9 Nov 2023cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Window attention, position embeddings, and high resolution finetuning are core concepts in the modern transformer era of computer vision. However, we find that naively combining these near ubiquitous components can have a detrimental effect on performance. The issue is simple: interpolating position embeddings while using window attention is wrong. We study two state-of-the-art methods that have these three components, namely Hiera and ViTDet, and find that both do indeed suffer from this bug. To fix it, we introduce a simple absolute window position embedding strategy, which solves the bug outright in Hiera and allows us to increase both speed and performance of the model in ViTDet. We finally combine the two to obtain HieraDet, which achieves 61.7 box mAP on COCO, making it state-of-the-art for models that only use ImageNet-1k pretraining. This all stems from what is essentially a 3 line bug fix, which we name "absolute win".

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 48 of its 52 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

48 of 52 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2023Segment Anythingcited by 8,158
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Mask R-CNNcited by 7,259
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
36 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  13. 2018Self-Attention with Relative Position Representationscited by 1,112
  14. 2021Scaling Vision Transformerscited by 993
  15. 2021Multiscale Vision Transformerscited by 896
  16. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  17. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  18. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  20. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  21. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  22. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  23. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  24. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  25. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  26. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  27. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  28. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  29. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  30. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  31. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  32. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  33. 2021CBNet: A Composite Backbone Network Architecture for Object Detectioncited by 77
  34. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  35. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  36. 2021LVIS Challenge Track Technical Report 1st Place Solution: Distribution Balanced and Boundary Refinement for Large Vocabulary Instance Segmentationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 9; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  4. 2024When Do We Not Need Larger Vision Models?cited by 65
  5. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 33
  6. 2025OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Promptscited by 14
  7. 2025MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentationcited by 9
  8. 2025Boltzmann Attention Sampling for Image Analysis with Small Objectscited by 4
  9. 2024From SAM to SAM 2: Exploring Improvements in Meta's Segment Anything Modelcited by 3
  10. 2025Fine-Grained Spatially Varying Material Selection in Imagescited by 3
  11. 2025SAM4D: Segment Anything in Camera and LiDAR Streamscited by 3
  12. 2025Segment Anything Across Shots: A Method and Benchmarkcited by 2
6 more
  1. 2026Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrievalcited by 2
  2. 2024Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentationcited by 0
  3. 2025DAM-Net: Domain Adaptation Network with Micro-Labeled Fine-Tuning for Change Detectioncited by 0
  4. 2025Automated urban waterlogging assessment and early warning through a mixture of foundation modelscited by 0
  5. 2026Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3)cited by 0
  6. 2026OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentationcited by 0