# One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression

arXiv: https://arxiv.org/abs/2501.10064 · HTML: https://mlsanity.com/paper/2501.10064/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2025.

## Built on by (most cited first, 30 of 48)

- [Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning](https://mlsanity.com/paper/2505.07538/) (2025)
- ["Principal Components" Enable A New Language of Images](https://mlsanity.com/paper/2503.08685/) (2025)
- [Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2507.08441/) (2025)
- [Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space](https://mlsanity.com/paper/2505.17011/) (2025)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [Single-pass Adaptive Image Tokenization for Minimum Program Search](https://mlsanity.com/paper/2507.07995/) (2025)
- [Image Tokenizer Needs Post-Training](https://mlsanity.com/paper/2509.12474/) (2025)
- [Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesis](https://mlsanity.com/paper/2503.08354/) (2025)
- [GPSToken: Gaussian Parameterized Spatially-adaptive Tokenization for Image Representation and Generation](https://mlsanity.com/paper/2509.01109/) (2025)
- [InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression](https://mlsanity.com/paper/2512.16975/) (2025)
- [Decoupling High and Low Frequencies for Faithful Image Generation with Fine Details](https://mlsanity.com/paper/2509.05441/) (2025)
- [VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generation](https://mlsanity.com/paper/2512.19680/) (2025)
- [EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation](https://mlsanity.com/paper/2603.12267/) (2026)
- [REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization](https://mlsanity.com/paper/2510.04450/) (2025)
- [Improving Flexible Image Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2601.01535/) (2026)
- [Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model](https://mlsanity.com/paper/2603.05438/) (2026)
- [VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization](https://mlsanity.com/paper/2604.12887/) (2026)
- [ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2601.03955/) (2026)
- [CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization](https://mlsanity.com/paper/2603.06449/) (2026)
- [Masked Generative Nested Transformers with Decode Time Scaling](https://mlsanity.com/paper/2502.00382/) (2025)
- [Generative Video Compression with One-Dimensional Latent Representation](https://mlsanity.com/paper/2603.15302/) (2026)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generation](https://mlsanity.com/paper/2512.21867/) (2025)
- [SFTok: Bridging the Performance Gap in Discrete Tokenizers](https://mlsanity.com/paper/2512.16910/) (2025)
- [Soft Tail-dropping for Adaptive Visual Tokenization](https://mlsanity.com/paper/2601.14246/) (2026)
- [Semantic-Aware Prefix Learning for Token-Efficient Image Generation](https://mlsanity.com/paper/2603.25249/) (2026)
- [(1D) Ordered Tokens Enable Efficient Test-Time Search](https://mlsanity.com/paper/2604.15453/) (2026)
- [VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generations](https://mlsanity.com/paper/2604.24885/) (2026)
- [Vision Foundation Models as Generalist Tokenizers for Image Generation](https://mlsanity.com/paper/2605.18390/) (2026)
- [Balancing Image Compression and Generation with Bootstrapped Tokenization](https://mlsanity.com/paper/2606.05552/) (2026)

## Abstract (arXiv)

Current image tokenization methods require a large number of tokens to capture the information contained within images. Although the amount of information varies across images, most image tokenizers only support fixed-length tokenization, leading to inefficiency in token allocation. In this study, we introduce One-D-Piece, a discrete image tokenizer designed for variable-length tokenization, achieving quality-controllable mechanism. To enable variable compression rate, we introduce a simple but effective regularization mechanism named "Tail Token Drop" into discrete one-dimensional image tokenizers. This method encourages critical information to concentrate at the head of the token sequence, enabling support of variadic tokenization, while preserving state-of-the-art reconstruction quality. We evaluate our tokenizer across multiple reconstruction quality metrics and find that it delivers significantly better perceptual quality than existing quality-controllable compression methods, including JPEG and WebP, at smaller byte sizes. Furthermore, we assess our tokenizer on various downstream computer vision tasks, including image classification, object detection, semantic segmentation, and depth estimation, confirming its adaptability to numerous applications compared to other variable-rate methods. Our approach demonstrates the versatility of variable-length discrete image tokenization, establishing a new paradigm in both compression efficiency and reconstruction performance. Finally, we validate the effectiveness of tail token drop via detailed analysis of tokenizers.