mlsanityData to 7 Oct 2026

LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate

arXiv 2405.13985 ↗First posted 22 May 2024cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

High-resolution images offer more information about scenes that can improve model accuracy. However, the dominant model architecture in computer vision, the vision transformer (ViT), cannot effectively leverage larger images without finetuning -- ViTs poorly extrapolate to more patches at test time, although transformers offer sequence length flexibility. We attribute this shortcoming to the current patch position encoding methods, which create a distribution shift when extrapolating. We propose a drop-in replacement for the position encoding of plain ViTs that restricts attention heads to fixed fields of view, pointed in different directions, using 2D attention masks. Our novel method, called LookHere, provides translation-equivariance, ensures attention head diversity, and limits the distribution shift that attention heads face when extrapolating. We demonstrate that LookHere improves performance on classification (avg. 1.6%), against adversarial attack (avg. 5.4%), and decreases calibration error (avg. 1.5%) -- on ImageNet without extrapolation. With extrapolation, LookHere outperforms the current SoTA position encoding method, 2D-RoPE, by 21.7% on ImageNet when trained at 2242 px and tested at 10242 px. Additionally, we release a high-resolution test set to improve the evaluation of high-resolution image classifiers, called ImageNet-HR.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 89 of its 108 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

89 of 108 references.

References · 89

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
38 more
  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2024Mixtral of Expertscited by 1,785
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  15. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  16. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  17. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  18. 2018Robustness May Be at Odds with Accuracycited by 873
  19. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  20. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  21. 2019Are Sixteen Heads Really Better than One?cited by 803
  22. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  23. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  24. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  25. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  26. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  27. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  28. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  29. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  30. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  31. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  32. 2022DeiT III: Revenge of the ViTcited by 417
  33. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  34. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  35. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  36. 2020Are we done with ImageNet?cited by 329
  37. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  38. 2017A Structured Self-attentive Sentence Embeddingcited by 318

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervisioncited by 5
  2. 2025Thicker and Quicker: A Jumbo Token for Fast Plain Vision Transformerscited by 4
  3. 2025LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encodingcited by 3
  4. 2025Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimizationcited by 1
  5. 2026Self-Soupervision: Cooking Model Soups without Labelscited by 1
  6. 2026Efficiency Follows Global-Local Decouplingcited by 1
  7. 2025Focus Your Attention: Towards Data-Intuitive Lightweight Vision Transformerscited by 0
  8. 2026Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, Generalcited by 0
  9. 2026LookThere! Sparse Vision by Reinforced Selectioncited by 0