# ViTAR: Vision Transformer with Any Resolution

arXiv: https://arxiv.org/abs/2403.18361 · HTML: https://mlsanity.com/paper/2403.18361/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [DaViT: Dual Attention Vision Transformers](https://mlsanity.com/paper/2204.03645/) (2022)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [All Tokens Matter: Token Labeling for Training Better Vision Transformers](https://mlsanity.com/paper/2104.10858/) (2021)
- [Randomized Positional Encodings Boost Length Generalization of Transformers](https://mlsanity.com/paper/2305.16843/) (2023)
- [Inception Transformer](https://mlsanity.com/paper/2205.12956/) (2022)
- [SVFormer: Semi-supervised Video Transformer for Action Recognition](https://mlsanity.com/paper/2211.13222/) (2022)

## Built on by (most cited first, 25 of 25)

- [Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution](https://mlsanity.com/paper/2409.12961/) (2024)
- [FlexiMo: A Flexible Remote Sensing Foundation Model](https://mlsanity.com/paper/2503.23844/) (2025)
- [EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation](https://mlsanity.com/paper/2503.15831/) (2025)
- [RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers](https://mlsanity.com/paper/2505.13344/) (2025)
- [PathoHR: Breast Cancer Survival Prediction on High-Resolution Pathological Images](https://mlsanity.com/paper/2503.17970/) (2025)
- [MATEY: multiscale adaptive foundation models for spatiotemporal physical systems](https://mlsanity.com/paper/2412.20601/) (2024)
- [Cross-Domain Foundation Model Adaptation: Pioneering Computer Vision Models for Geophysical Data Analysis](https://mlsanity.com/paper/2408.12396/) (2024)
- [UniViTAR: Unified Vision Transformer with Native Resolution](https://mlsanity.com/paper/2504.01792/) (2025)
- [SeqPE: Transformer with Sequential Position Encoding](https://mlsanity.com/paper/2506.13277/) (2025)
- [Tile-Based ViT Inference with Visual-Cluster Priors for Zero-Shot Multi-Species Plant Identification](https://mlsanity.com/paper/2507.06093/) (2025)
- [Cross-Scale Pansharpening via ScaleFormer and the PanScale Benchmark](https://mlsanity.com/paper/2603.00543/) (2026)
- [Liveness Detection in Computer Vision: Transformer-based Self-Supervised Learning for Face Anti-Spoofing](https://mlsanity.com/paper/2406.13860/) (2024)
- [SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation](https://mlsanity.com/paper/2409.04082/) (2024)
- [MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy](https://mlsanity.com/paper/2602.24222/) (2026)
- [VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generations](https://mlsanity.com/paper/2604.24885/) (2026)
- [G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity](https://mlsanity.com/paper/2609.15018/) (2026)
- [Input Resolution Downsizing as a Compression Technique for Vision Deep Learning Systems](https://mlsanity.com/paper/2504.03749/) (2025)
- [StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scales](https://mlsanity.com/paper/2510.16209/) (2025)
- [ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers](https://mlsanity.com/paper/2604.23798/) (2026)
- [On What We Can Learn from Low-Resolution Data](https://mlsanity.com/paper/2605.12168/) (2026)
- [Weighted Reverse Convolution for Feature Upsampling](https://mlsanity.com/paper/2605.17472/) (2026)
- [Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers](https://mlsanity.com/paper/2607.00580/) (2026)
- [PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks](https://mlsanity.com/paper/2608.02792/) (2026)
- [Variable-Granularity Tokenization for High-Resolution Object Detection](https://mlsanity.com/paper/2608.28706/) (2026)
- [ProgResViT: Progressive Resolution and Width for Adaptive Vision Transformers](https://mlsanity.com/paper/2609.03216/) (2026)

## Abstract (arXiv)

This paper tackles a significant challenge faced by Vision Transformers (ViTs): their constrained scalability across different image resolutions. Typically, ViTs experience a performance decline when processing resolutions different from those seen during training. Our work introduces two key innovations to address this issue. Firstly, we propose a novel module for dynamic resolution adjustment, designed with a single Transformer block, specifically to achieve highly efficient incremental token integration. Secondly, we introduce fuzzy positional encoding in the Vision Transformer to provide consistent positional awareness across multiple resolutions, thereby preventing overfitting to any single training resolution. Our resulting model, ViTAR (Vision Transformer with Any Resolution), demonstrates impressive adaptability, achieving 83.3% top-1 accuracy at a 1120x1120 resolution and 80.4% accuracy at a 4032x4032 resolution, all while reducing computational costs. ViTAR also shows strong performance in downstream tasks such as instance and semantic segmentation and can easily combined with self-supervised learning techniques like Masked AutoEncoder. Our work provides a cost-effective solution for enhancing the resolution scalability of ViTs, paving the way for more versatile and efficient high-resolution image processing.