# MaskBit: Embedding-free Image Generation via Bit Tokens

arXiv: https://arxiv.org/abs/2409.16211 · HTML: https://mlsanity.com/paper/2409.16211/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 50 of its 64 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)

## Built on by (most cited first, 30 of 98)

- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [Randomized Autoregressive Visual Generation](https://mlsanity.com/paper/2411.00776/) (2024)
- [Masked Autoencoders Are Effective Tokenizers for Diffusion Models](https://mlsanity.com/paper/2502.03444/) (2025)
- [ImageFolder: Autoregressive Image Generation with Folded Tokens](https://mlsanity.com/paper/2410.01756/) (2024)
- [RandAR: Decoder-only Autoregressive Visual Generation in Random Orders](https://mlsanity.com/paper/2412.01827/) (2024)
- [FlowAR: Scale-wise Autoregressive Image Generation Meets Flow Matching](https://mlsanity.com/paper/2412.15205/) (2024)
- [Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2502.20388/) (2025)
- [GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation](https://mlsanity.com/paper/2504.08736/) (2025)
- [SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer](https://mlsanity.com/paper/2412.10958/) (2024)
- [Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens](https://mlsanity.com/paper/2501.07730/) (2025)
- [ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer](https://mlsanity.com/paper/2412.07720/) (2024)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [FlowTok: Flowing Seamlessly Across Text and Image Tokens](https://mlsanity.com/paper/2503.10772/) (2025)
- [Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning](https://mlsanity.com/paper/2505.07538/) (2025)
- [Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2412.18619/) (2024)
- ["Principal Components" Enable A New Language of Images](https://mlsanity.com/paper/2503.08685/) (2025)
- [Next Patch Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15321/) (2024)
- [Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator](https://mlsanity.com/paper/2503.01103/) (2025)
- [Autoregressive Image Generation with Randomized Parallel Decoding](https://mlsanity.com/paper/2503.10568/) (2025)
- [HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation](https://mlsanity.com/paper/2506.04421/) (2025)
- [ReDi: Rectified Discrete Flow](https://mlsanity.com/paper/2507.15897/) (2025)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [TokBench: Evaluating Your Visual Tokenizer before Visual Generation](https://mlsanity.com/paper/2505.18142/) (2025)
- [DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer](https://mlsanity.com/paper/2507.04947/) (2025)
- [Progressive Compositionality in Text-to-Image Generative Models](https://mlsanity.com/paper/2410.16719/) (2024)
- [XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generation](https://mlsanity.com/paper/2412.01762/) (2024)
- [MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization](https://mlsanity.com/paper/2504.00999/) (2025)
- [Image Tokenizer Needs Post-Training](https://mlsanity.com/paper/2509.12474/) (2025)
- [[MASK] is All You Need](https://mlsanity.com/paper/2412.06787/) (2024)
- [Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective](https://mlsanity.com/paper/2410.22217/) (2024)

## Abstract (arXiv)

Masked transformer models for class-conditional image generation have become a compelling alternative to diffusion models. Typically comprising two stages - an initial VQGAN model for transitioning between latent space and image space, and a subsequent Transformer model for image generation within latent space - these frameworks offer promising avenues for image synthesis. In this study, we present two primary contributions: Firstly, an empirical and systematic examination of VQGANs, leading to a modernized VQGAN. Secondly, a novel embedding-free generation network operating directly on bit tokens - a binary quantized representation of tokens with rich semantics. The first contribution furnishes a transparent, reproducible, and high-performing VQGAN model, enhancing accessibility and matching the performance of current state-of-the-art methods while revealing previously undisclosed details. The second contribution demonstrates that embedding-free image generation using bit tokens achieves a new state-of-the-art FID of 1.52 on the ImageNet 256x256 benchmark, with a compact generator model of mere 305M parameters. The code for this project is available on https://github.com/markweberdev/maskbit.