# Vision Transformers with Mixed-Resolution Tokenization

arXiv: https://arxiv.org/abs/2304.00287 · HTML: https://mlsanity.com/paper/2304.00287/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 1 Apr 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)

## Built on by (most cited first, 30 of 36)

- [Describe Anything: Detailed Localized Image and Video Captioning](https://mlsanity.com/paper/2504.16072/) (2025)
- [Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance](https://mlsanity.com/paper/2403.06265/) (2024)
- [Subobject-level Image Tokenization](https://mlsanity.com/paper/2402.14327/) (2024)
- [CAT: Content-Adaptive Image Tokenization](https://mlsanity.com/paper/2501.03120/) (2025)
- [MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers](https://mlsanity.com/paper/2307.02321/) (2023)
- [SAT-HMR: Real-Time Multi-Person 3D Mesh Estimation via Scale-Adaptive Tokens](https://mlsanity.com/paper/2411.19824/) (2024)
- [Componential Prompt-Knowledge Alignment for Domain Incremental Learning](https://mlsanity.com/paper/2505.04575/) (2025)
- [Accelerating Vision Transformers with Adaptive Patch Sizes](https://mlsanity.com/paper/2510.18091/) (2025)
- [Wavelet-Based Image Tokenizer for Vision Transformers](https://mlsanity.com/paper/2405.18616/) (2024)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens](https://mlsanity.com/paper/2412.04680/) (2024)
- [Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment](https://mlsanity.com/paper/2504.02522/) (2025)
- [Fine-grained Image Aesthetic Assessment: Learning Discriminative Scores from Relative Ranks](https://mlsanity.com/paper/2603.03907/) (2026)
- [Differentiable Hierarchical Visual Tokenization](https://mlsanity.com/paper/2511.02652/) (2025)
- [DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.16968/) (2026)
- [Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation](https://mlsanity.com/paper/2603.23491/) (2026)
- [Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers](https://mlsanity.com/paper/2309.13353/) (2023)
- [SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling](https://mlsanity.com/paper/2508.15190/) (2025)
- [Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representations](https://mlsanity.com/paper/2403.07887/) (2024)
- [Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding](https://mlsanity.com/paper/2403.18593/) (2024)
- [ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutions](https://mlsanity.com/paper/2407.08691/) (2024)
- [Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation](https://mlsanity.com/paper/2411.10281/) (2024)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction](https://mlsanity.com/paper/2603.26258/) (2026)
- [Multiple-Resolution Tokenization for Time Series Forecasting with an Application to Pricing](https://mlsanity.com/paper/2407.03185/) (2024)
- [Adaptive Volumetric Mechanical Property Fields Invariant to Resolution](https://mlsanity.com/paper/2606.18231/) (2026)
- [Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions](https://mlsanity.com/paper/2509.14165/) (2025)
- [ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformers](https://mlsanity.com/paper/2601.21362/) (2026)
- [Token Warping Helps MLLMs Look from Nearby Viewpoints](https://mlsanity.com/paper/2604.02870/) (2026)
- [Adaptive Patching Is Harder Than It Looks For Time-Series Forecasting](https://mlsanity.com/paper/2606.04074/) (2026)

## Abstract (arXiv)

Vision Transformer models process input images by dividing them into a spatially regular grid of equal-size patches. Conversely, Transformers were originally introduced over natural language sequences, where each token represents a subword - a chunk of raw data of arbitrary size. In this work, we apply this approach to Vision Transformers by introducing a novel image tokenization scheme, replacing the standard uniform grid with a mixed-resolution sequence of tokens, where each token represents a patch of arbitrary size. Using the Quadtree algorithm and a novel saliency scorer, we construct a patch mosaic where low-saliency areas of the image are processed in low resolution, routing more of the model's capacity to important image regions. Using the same architecture as vanilla ViTs, our Quadformer models achieve substantial accuracy gains on image classification when controlling for the computational budget. Code and models are publicly available at https://github.com/TomerRonen34/mixed-resolution-vit .