# An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels

arXiv: https://arxiv.org/abs/2406.09415 · HTML: https://mlsanity.com/paper/2406.09415/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 74 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)

## Built on by (most cited first, 30 of 33)

- [Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More](https://mlsanity.com/paper/2502.03738/) (2025)
- [The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding](https://mlsanity.com/paper/2512.19693/) (2025)
- [NeurIPT: Foundation Model for Neural Interfaces](https://mlsanity.com/paper/2510.16548/) (2025)
- [Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency](https://mlsanity.com/paper/2410.07599/) (2024)
- [Imbalance-Aware Culvert-Sewer Defect Segmentation Using an Enhanced Feature Pyramid Network](https://mlsanity.com/paper/2408.10181/) (2024)
- [Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval](https://mlsanity.com/paper/2505.15877/) (2025)
- [Quantum Hamiltonian Embedding of Images for Data Reuploading Classifiers](https://mlsanity.com/paper/2407.14055/) (2024)
- [SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Images](https://mlsanity.com/paper/2508.05202/) (2025)
- [RadioFormer: A Multiple-Granularity Radio Map Estimation Transformer with 1textpertenthousand Spatial Sampling](https://mlsanity.com/paper/2504.19161/) (2025)
- [Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few](https://mlsanity.com/paper/2509.16875/) (2025)
- [Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?](https://mlsanity.com/paper/2511.08704/) (2025)
- [ECHOSAT: Estimating Canopy Height Over Space And Time](https://mlsanity.com/paper/2602.21421/) (2026)
- [Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusion](https://mlsanity.com/paper/2606.15236/) (2026)
- [Hierarchical Compact Clustering Attention (COCA) for Unsupervised Object-Centric Learning](https://mlsanity.com/paper/2505.02071/) (2025)
- [GViT: Representing Images as Gaussians for Visual Recognition](https://mlsanity.com/paper/2506.23532/) (2025)
- [Assessing the value of Geo-Foundational Models for Flood Inundation Mapping: Benchmarking models for Sentinel-1, Sentinel-2, and Planetscope for end-users](https://mlsanity.com/paper/2511.01990/) (2025)
- [DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views](https://mlsanity.com/paper/2601.15516/) (2026)
- [Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability](https://mlsanity.com/paper/2603.04807/) (2026)
- [A Decade of Deep Learning: A Survey on The Magnificent Seven](https://mlsanity.com/paper/2412.16188/) (2024)
- [A Study on Context Length and Efficient Transformers for Biomedical Image Analysis](https://mlsanity.com/paper/2501.00619/) (2024)
- [Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing](https://mlsanity.com/paper/2502.00594/) (2025)
- [SPoT: Subpixel Placement of Tokens in Vision Transformers](https://mlsanity.com/paper/2507.01654/) (2025)
- [MCGA: Mixture of Codebooks Hyperspectral Reconstruction via Grayscale-Aware Attention](https://mlsanity.com/paper/2507.09885/) (2025)
- [LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation](https://mlsanity.com/paper/2507.18940/) (2025)
- [A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports](https://mlsanity.com/paper/2508.13796/) (2025)
- [Uncertainty in Semantic Language Modeling with PIXELS](https://mlsanity.com/paper/2509.19563/) (2025)
- [Monocular Visual Odometry without Calibration or Test-time Optimization](https://mlsanity.com/paper/2510.03348/) (2025)
- [Interpretable and Sparse Linear Attention with Decoupled Membership-Subspace Modeling via MCR2 Objective](https://mlsanity.com/paper/2601.17042/) (2026)
- [Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification](https://mlsanity.com/paper/2602.18614/) (2026)
- [Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers](https://mlsanity.com/paper/2606.14757/) (2026)

## Abstract (arXiv)

This work does not introduce a new method. Instead, we present an interesting finding that questions the necessity of the inductive bias of locality in modern computer vision architectures. Concretely, we find that vanilla Transformers can operate by directly treating each individual pixel as a token and achieve highly performant results. This is substantially different from the popular design in Vision Transformer, which maintains the inductive bias from ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a token). We showcase the effectiveness of pixels-as-tokens across three well-studied computer vision tasks: supervised learning for classification and regression, self-supervised learning via masked autoencoding, and image generation with diffusion models. Although it's computationally less practical to directly operate on individual pixels, we believe the community must be made aware of this surprising piece of knowledge when devising the next generation of neural network architectures for computer vision.