# Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers

arXiv: https://arxiv.org/abs/2309.13353 · HTML: https://mlsanity.com/paper/2309.13353/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 23 Sep 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)
- [A Multigrid Method for Efficiently Training Video Models](https://mlsanity.com/paper/1912.00998/) (2019)
- [TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers](https://mlsanity.com/paper/2207.08409/) (2022)
- [Vision Transformers with Mixed-Resolution Tokenization](https://mlsanity.com/paper/2304.00287/) (2023)
- [PatchDropout: Economizing Vision Transformers Using Patch Dropout](https://mlsanity.com/paper/2208.07220/) (2022)
- [MVT: Mask Vision Transformer for Facial Expression Recognition in the wild](https://mlsanity.com/paper/2106.04520/) (2021)

## Built on by (most cited first, 3 of 3)

- [AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale](https://mlsanity.com/paper/2404.03482/) (2024)
- [SPoT: Subpixel Placement of Tokens in Vision Transformers](https://mlsanity.com/paper/2507.01654/) (2025)
- [Efficient Multi-Source Knowledge Transfer by Model Merging](https://mlsanity.com/paper/2508.19353/) (2025)

## Abstract (arXiv)

Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.