# "Principal Components" Enable A New Language of Images

arXiv: https://arxiv.org/abs/2503.08685 · HTML: https://mlsanity.com/paper/2503.08685/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 67 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)

## Built on by (most cited first, 27 of 27)

- [ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code](https://mlsanity.com/paper/2506.02314/) (2025)
- [AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models](https://mlsanity.com/paper/2509.25162/) (2025)
- [Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2507.08441/) (2025)
- [SceneTok: A Compressed, Diffusable Token Space for 3D Scenes](https://mlsanity.com/paper/2602.18882/) (2026)
- [VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization](https://mlsanity.com/paper/2604.12887/) (2026)
- [ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2601.03955/) (2026)
- [LoST: Level of Semantics Tokenization for 3D Shapes](https://mlsanity.com/paper/2603.17995/) (2026)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [Ordered Action Tokens for Visuomotor Policy Learning](https://mlsanity.com/paper/2607.21670/) (2026)
- [Decoupling Complexity from Scale in Latent Diffusion Model](https://mlsanity.com/paper/2511.16117/) (2025)
- [Composable Visual Tokenizers with Generator-Free Diagnostics of Learnability](https://mlsanity.com/paper/2602.03339/) (2026)
- [(1D) Ordered Tokens Enable Efficient Test-Time Search](https://mlsanity.com/paper/2604.15453/) (2026)
- [Vision Foundation Models as Generalist Tokenizers for Image Generation](https://mlsanity.com/paper/2605.18390/) (2026)
- [PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding](https://mlsanity.com/paper/2605.30126/) (2026)
- [AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens](https://mlsanity.com/paper/2606.07185/) (2026)
- [AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation](https://mlsanity.com/paper/2606.30811/) (2026)
- [Motion Planning in Compressed Representation Spaces](https://mlsanity.com/paper/2606.30940/) (2026)
- [ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes](https://mlsanity.com/paper/2609.01740/) (2026)
- [RAC: Rectified Flow Auto Coder](https://mlsanity.com/paper/2603.05925/) (2026)
- [Channel-wise Vector Quantization](https://mlsanity.com/paper/2605.26089/) (2026)
- [Structure over Pixels: Learning Variable-Length Visual Programs](https://mlsanity.com/paper/2605.27696/) (2026)
- [Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers](https://mlsanity.com/paper/2606.20076/) (2026)
- [On the Diffusibility of High-Dimensional Latents](https://mlsanity.com/paper/2609.28473/) (2026)
- [NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generation](https://mlsanity.com/paper/2609.36756/) (2026)
- [SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation](https://mlsanity.com/paper/2610.00686/) (2026)
- [Semantically Steerable Image Tokenization](https://mlsanity.com/paper/2610.15515/) (2026)
- [Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels](https://mlsanity.com/paper/2610.16552/) (2026)

## Abstract (arXiv)

We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space. While existing visual tokenizers primarily optimize for reconstruction fidelity, they often neglect the structural properties of the latent space--a critical factor for both interpretability and downstream tasks. Our method generates a 1D causal token sequence for images, where each successive token contributes non-overlapping information with mathematically guaranteed decreasing explained variance, analogous to principal component analysis. This structural constraint ensures the tokenizer extracts the most salient visual features first, with each subsequent token adding diminishing yet complementary information. Additionally, we identified and resolved a semantic-spectrum coupling effect that causes the unwanted entanglement of high-level semantic content and low-level spectral details in the tokens by leveraging a diffusion decoder. Experiments demonstrate that our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system. Moreover, autoregressive models trained on our token sequences achieve performance comparable to current state-of-the-art methods while requiring fewer tokens for training and inference.