# Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens

arXiv: https://arxiv.org/abs/2501.07730 · HTML: https://mlsanity.com/paper/2501.07730/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 72 references that are in the map.
- First posted on arXiv 13 Jan 2025.

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)

## Built on by (most cited first, 30 of 50)

- [REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers](https://mlsanity.com/paper/2504.10483/) (2025)
- [Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2502.20388/) (2025)
- [UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing](https://mlsanity.com/paper/2507.23278/) (2025)
- [FlowTok: Flowing Seamlessly Across Text and Image Tokens](https://mlsanity.com/paper/2503.10772/) (2025)
- [AToken: A Unified Tokenizer for Vision](https://mlsanity.com/paper/2509.14476/) (2025)
- [Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model](https://mlsanity.com/paper/2506.05289/) (2025)
- [Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation](https://mlsanity.com/paper/2510.13418/) (2025)
- [Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers](https://mlsanity.com/paper/2505.14687/) (2025)
- [DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer](https://mlsanity.com/paper/2507.04947/) (2025)
- [Image Tokenizer Needs Post-Training](https://mlsanity.com/paper/2509.12474/) (2025)
- [Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesis](https://mlsanity.com/paper/2503.08354/) (2025)
- [A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens](https://mlsanity.com/paper/2604.04913/) (2026)
- [Deeply Supervised Flow-Based Generative Models](https://mlsanity.com/paper/2503.14494/) (2025)
- [Laminating Representation Autoencoders for Efficient Diffusion](https://mlsanity.com/paper/2602.04873/) (2026)
- [Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings](https://mlsanity.com/paper/2509.22925/) (2025)
- [Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model](https://mlsanity.com/paper/2512.22288/) (2025)
- [Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model](https://mlsanity.com/paper/2603.05438/) (2026)
- [DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignment](https://mlsanity.com/paper/2603.22125/) (2026)
- [Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation](https://mlsanity.com/paper/2506.21022/) (2025)
- [PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation](https://mlsanity.com/paper/2602.15882/) (2026)
- [CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization](https://mlsanity.com/paper/2603.06449/) (2026)
- [Generative Video Compression with One-Dimensional Latent Representation](https://mlsanity.com/paper/2603.15302/) (2026)
- [Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation](https://mlsanity.com/paper/2605.25195/) (2026)
- [Layton: Latent Consistency Tokenizer for 1024-pixel Image Reconstruction and Generation by 256 Tokens](https://mlsanity.com/paper/2503.08377/) (2025)
- [FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens](https://mlsanity.com/paper/2506.03096/) (2025)
- [Plan-X: Instruct Video Generation via Semantic Planning](https://mlsanity.com/paper/2511.17986/) (2025)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models](https://mlsanity.com/paper/2604.18518/) (2026)
- [TaskTok: Delving into Task Tokens for Task-driven Image Restoration](https://mlsanity.com/paper/2606.26615/) (2026)
- [Aligning Latent Spaces with Flow Priors](https://mlsanity.com/paper/2506.05240/) (2025)

## Abstract (arXiv)

Image tokenizers form the foundation of modern text-to-image generative models but are notoriously difficult to train. Furthermore, most existing text-to-image models rely on large-scale, high-quality private datasets, making them challenging to replicate. In this work, we introduce Text-Aware Transformer-based 1-Dimensional Tokenizer (TA-TiTok), an efficient and powerful image tokenizer that can utilize either discrete or continuous 1-dimensional tokens. TA-TiTok uniquely integrates textual information during the tokenizer decoding stage (i.e., de-tokenization), accelerating convergence and enhancing performance. TA-TiTok also benefits from a simplified, yet effective, one-stage training process, eliminating the need for the complex two-stage distillation used in previous 1-dimensional tokenizers. This design allows for seamless scalability to large datasets. Building on this, we introduce a family of text-to-image Masked Generative Models (MaskGen), trained exclusively on open data while achieving comparable performance to models trained on private data. We aim to release both the efficient, strong TA-TiTok tokenizers and the open-data, open-weight MaskGen models to promote broader access and democratize the field of text-to-image masked generative models.