# Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More

arXiv: https://arxiv.org/abs/2502.03738 · HTML: https://mlsanity.com/paper/2502.03738/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 71 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 30 of 63)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)

## Built on by (most cited first, 28 of 28)

- [Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail](https://mlsanity.com/paper/2511.00088/) (2025)
- [FlexiMo: A Flexible Remote Sensing Foundation Model](https://mlsanity.com/paper/2503.23844/) (2025)
- [OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning](https://mlsanity.com/paper/2505.04601/) (2025)
- [Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning](https://mlsanity.com/paper/2511.19515/) (2025)
- [ViT-5: Vision Transformers for The Mid-2020s](https://mlsanity.com/paper/2602.08071/) (2026)
- [Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving](https://mlsanity.com/paper/2506.12251/) (2025)
- [DeepScan: A Training-Free Framework for Visually Grounded Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2603.03857/) (2026)
- [ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models](https://mlsanity.com/paper/2504.00037/) (2025)
- [Overtone: Cyclic Patch Modulation for Clean, Efficient, and Flexible Physics Emulators](https://mlsanity.com/paper/2507.09264/) (2025)
- [(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models](https://mlsanity.com/paper/2604.16429/) (2026)
- [HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models](https://mlsanity.com/paper/2505.20444/) (2025)
- [Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few](https://mlsanity.com/paper/2509.16875/) (2025)
- [Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?](https://mlsanity.com/paper/2511.08704/) (2025)
- [AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens](https://mlsanity.com/paper/2511.18105/) (2025)
- [RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models](https://mlsanity.com/paper/2511.18380/) (2025)
- [DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views](https://mlsanity.com/paper/2601.15516/) (2026)
- [Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability](https://mlsanity.com/paper/2603.04807/) (2026)
- [TC-AE: Unlocking Token Capacity for Deep Compression Autoencoders](https://mlsanity.com/paper/2604.07340/) (2026)
- [Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density](https://mlsanity.com/paper/2605.17340/) (2026)
- [Improving MLLM Historical Record Extraction with Test-Time Image](https://mlsanity.com/paper/2509.09722/) (2025)
- [Neural Networks for Predicting Permeability Tensors of 2D Porous Media: Comparison of Convolution- and Transformer-based Architectures](https://mlsanity.com/paper/2512.01517/) (2025)
- [Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification](https://mlsanity.com/paper/2602.18614/) (2026)
- [Small Models, Strong Priors: Architectural Inductive Bias for Parameter-Efficient Neural PDE Solvers](https://mlsanity.com/paper/2605.25949/) (2026)
- [RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video](https://mlsanity.com/paper/2605.31535/) (2026)
- [Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers](https://mlsanity.com/paper/2606.14757/) (2026)
- [Now We Know? A Systematic Comparison of TerraMind and THOR](https://mlsanity.com/paper/2607.18504/) (2026)
- [Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions](https://mlsanity.com/paper/2607.19378/) (2026)
- [Pixel-Level Transformers in Remote Sensing: A Canopy Height Case Study](https://mlsanity.com/paper/2609.37809/) (2026)

## Abstract (arXiv)

Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a de facto image tokenization approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models. Code is available at https://github.com/wangf3014/Patch_Scaling.