mlsanityData to 7 Oct 2026

Masked Autoencoders Are Effective Tokenizers for Diffusion Models

arXiv 2502.03444 ↗First posted 5 Feb 2025cs.CV cs.AI cs.LGcited by 93

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Recent advances in latent diffusion models have demonstrated their effectiveness for high-resolution image synthesis. However, the properties of the latent space from tokenizer for better learning and generation of diffusion models remain under-explored. Theoretically and empirically, we find that improved generation quality is closely tied to the latent distributions with better structure, such as the ones with fewer Gaussian Mixture modes and more discriminative features. Motivated by these insights, we propose MAETok, an autoencoder (AE) leveraging mask modeling to learn semantically rich latent space while maintaining reconstruction fidelity. Extensive experiments validate our analysis, demonstrating that the variational form of autoencoders is not necessary, and a discriminative latent space from AE alone enables state-of-the-art performance on ImageNet generation using only 128 tokens. MAETok achieves significant practical improvements, enabling a gFID of 1.69 with 76x faster training and 31x higher inference throughput for 512x512 generation. Our findings show that the structure of the latent space, rather than variational constraints, is crucial for effective diffusion models. Code and trained models are released.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 72 of its 88 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

72 of 88 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  2. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models14 Oct
  3. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  4. 2024ImageFolder: Autoregressive Image Generation with Folded Tokens2 Oct
  5. 2024MaskBit: Embedding-free Image Generation via Bit Tokens24 Sep
  6. 2024CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models12 Jun
  7. 2024Guiding a Diffusion Model with a Bad Version of Itself4 Jun
  8. 2024Learning Mixtures of Gaussians Using Diffusion Models29 Apr
  9. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models11 Apr
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  11. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization7 Aug
  12. 2023Learning Mixtures of Gaussians Using the DDPM Objective3 Jul
11 more
  1. 2022Scalable Diffusion Models with Transformers19 Dec
  2. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions3 Nov
  3. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  4. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  5. 2022Classifier-Free Diffusion Guidance26 Jul
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  7. 2021Score-based Generative Modeling in Latent Space10 Jun
  8. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  9. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  10. 2020Denoising Diffusion Implicit Models6 Oct
  11. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
37 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  8. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  10. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  11. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  12. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  13. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  14. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  15. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  16. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  17. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  18. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  19. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  20. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  21. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  22. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 221
  23. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  24. 2024Randomized Autoregressive Visual Generationcited by 125
  25. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 105
  26. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  27. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  28. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  29. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  30. 2024Language-Guided Image Tokenization for Generationcited by 36
  31. 2022Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesiscited by 31
  32. 2024Causal Diffusion Transformers for Generative Modelingcited by 31
  33. 2024Scalable Autoregressive Image Generation with Mambacited by 26
  34. 2024Customize Your Visual Autoregressive Recipe with Set Autoregressive Modelingcited by 21
  35. 2025Visual Generation Without Guidancecited by 17
  36. 2024XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generationcited by 12
  37. 2024Classification Done Right for Vision-Language Pre-Trainingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 46; 2026: 47.

93 papers.

Sort:

By area:

  1. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  2. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  3. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  4. 2025Training Agents Inside of Scalable World Modelscited by 122
  5. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 75
  6. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  7. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50
  8. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Spacecited by 44
  9. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  10. 2025AToken: A Unified Tokenizer for Visioncited by 30
  11. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 27
  12. 2025"Principal Components" Enable A New Language of Imagescited by 25
38 more
  1. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 25
  2. 2026Improved Baselines with Representation Autoencoderscited by 24
  3. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  4. 2025Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulationcited by 19
  5. 2025Latent Denoising Makes Good Tokenizerscited by 18
  6. 2025DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizercited by 13
  7. 2025H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Modelscited by 12
  8. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 12
  9. 2025Physics-Informed Distillation of Diffusion Models for PDE-Constrained Generationcited by 11
  10. 2025Latent Diffusion Models with Masked AutoEncoderscited by 11
  11. 2025Image Tokenizer Needs Post-Trainingcited by 10
  12. 2025Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusabilitycited by 10
  13. 2025Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesiscited by 9
  14. 2025WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstructioncited by 9
  15. 2026RAE-NWM: Navigation World Model in Dense Visual Representation Spacecited by 9
  16. 2026Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matchingcited by 8
  17. 2025GPSToken: Gaussian Parameterized Spatially-adaptive Tokenization for Image Representation and Generationcited by 7
  18. 2025Decoupling High and Low Frequencies for Faithful Image Generation with Fine Detailscited by 7
  19. 2025HieraTok: Multi-Scale Visual Tokenizer Improves Image Reconstruction and Generationcited by 7
  20. 2025DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Spacecited by 7
  21. 2025SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenizationcited by 7
  22. 2026Hyperspherical Autoencoder for High-Fidelity Image Reconstruction and Generationcited by 7
  23. 2026What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusioncited by 7
  24. 2025MeanFlow Transformers with Representation Autoencoderscited by 6
  25. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 6
  26. 2025SemanticGen: Video Generation in Semantic Spacecited by 6
  27. 2025Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generationcited by 5
  28. 2025Scalable GANs with Transformerscited by 5
  29. 2025REGLUE Your Latents with Global and Local Semantics for Entangled Diffusioncited by 5
  30. 2026Improving Flexible Image Tokenizers for Autoregressive Image Generationcited by 5
  31. 2025VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumptioncited by 4
  32. 2025Rotary Masked Autoencoders are Versatile Learnerscited by 4
  33. 2025Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoderscited by 4
  34. 2025RecTok: Reconstruction Distillation along Rectified Flowcited by 4
  35. 2025Is There a Better Source Distribution than Gaussian? Exploring Source Distributions for Image Flow Matchingcited by 4
  36. 2026Boosting Latent Diffusion Models via Disentangled Representation Alignmentcited by 4
  37. 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Codecited by 4
  38. 2026DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignmentcited by 4

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesis
  2. 2024Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective
  3. 2026End-to-End Training for Unified Tokenization and Latent Denoising
  4. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
  5. 2023Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models
  6. 2023Hierarchical Diffusion Autoencoders and Disentangled Image Manipulation
  7. 2024Boosting Latent Diffusion with Perceptual Objectives
  8. 2024Nested Diffusion Models Using Hierarchical Latent Priors