Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers
Not on the map. Why?
Abstract (arXiv)
Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 23 Sep 2023.
Builds on
32 of 36 references.
References · 32
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2023Segment Anythingcited by 8,234
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
20 more
- 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
- 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
- 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
- 2022DeiT III: Revenge of the ViTcited by 419
- 2021Conditional Positional Encodings for Vision Transformerscited by 406
- 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
- 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
- 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
- 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
- 2022FlexiViT: One Model for All Patch Sizescited by 152
- 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
- 2021Patch Slimming for Efficient Vision Transformerscited by 133
- 2019A Multigrid Method for Efficiently Training Video Modelscited by 52
- 2022TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformerscited by 48
- 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 36
- 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
- 2021MVT: Mask Vision Transformer for Facial Expression Recognition in the wildcited by 27
- 2022Auto-scaling Vision Transformers without Trainingcited by 15
- 2023Active Visual Exploration Based on Attention-Map Entropycited by 13
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.