Vision encoders should be image size agnostic and task driven
Not on the map. Why?
Abstract (arXiv)
This position paper argues that the next generation of vision encoders should be image size agnostic and task driven. The source of our inspiration is biological. Not a structural aspect of biological vision, but a behavioral trait -- efficiency. We focus on a couple of ways in which vision in nature is efficient, but modern vision encoders not. We -- humans and animals -- deal with vast quantities of visual data, and need to be smart where we focus our limited energy -- it depends on the task. It is our belief that vision encoders should be dynamic and the computational complexity should depend on the task at hand rather than the size of the image. We, also, provide concrete first steps towards our vision -- a proof-of-concept solution for image classification. Despite classification being not very representative for what we are trying to achieve, it shows that our approach is feasible and promising.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 22 Aug 2025.
Builds on
33 of 49 references.
References · 33
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
21 more
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2014Going Deeper with Convolutionscited by 4,241
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
- 2021Multiscale Vision Transformerscited by 897
- 2023Vision Transformers Need Registerscited by 871
- 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
- 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
- 2021Neural Fields in Visual Computing and Beyondcited by 540
- 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
- 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
- 2014Recurrent Models of Visual Attentioncited by 349
- 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
- 2019Fixing the train-test resolution discrepancycited by 209
- 2022Recurrent Memory Transformercited by 203
- 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
- 2020GMAT: Global Memory Augmentation for Transformerscited by 36
- 2021FoveaTer: Foveated Transformer for Image Classificationcited by 28
- 2020Unsupervised Foveal Vision Neural Networks with Top-Down Attentioncited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.