mlsanityData to 14 Oct 2026

Latent Diffusion Models with Masked AutoEncoders

arXiv 2507.09984 ↗First posted 14 Jul 2025cs.CVcited by 11

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In spite of the remarkable potential of Latent Diffusion Models (LDMs) in image generation, the desired properties and optimal design of the autoencoders have been underexplored. In this work, we analyze the role of autoencoders in LDMs and identify three key properties: latent smoothness, perceptual compression quality, and reconstruction quality. We demonstrate that existing autoencoders fail to simultaneously satisfy all three properties, and propose Variational Masked AutoEncoders (VMAEs), taking advantage of the hierarchical features maintained by Masked AutoEncoders. We integrate VMAEs into the LDM framework, introducing Latent Diffusion Models with Masked AutoEncoders (LDMAEs). Our code is available at https://github.com/isno0907/ldmae.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 39 of its 47 references that are in the map. First posted on arXiv 14 Jul 2025.

Builds on

39 of 47 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Models5 Feb
  2. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  3. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models14 Oct
  4. 2024Isometric Representation Learning for Disentangled Latent Space of Diffusion Models16 Jul
  5. 2024LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models23 May
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  7. 2023Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing30 Nov
  8. 2022Scalable Diffusion Models with Transformers19 Dec
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  11. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  12. 2021Diffusion Models Beat GANs on Image Synthesis11 May
4 more
  1. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  3. 2020Denoising Diffusion Implicit Models6 Oct
  4. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 23

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  10. 2016Improved Techniques for Training GANscited by 2,970
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
11 more
  1. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  3. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  4. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 628
  5. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  6. 2021Generating Images with Sparse Representationscited by 266
  7. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  8. 2023Delta Denoising Scorecited by 121
  9. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  10. 2021Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANscited by 26
  11. 2024Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  2. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 13
  3. 2025Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspectivecited by 10
  4. 2025Distribution Matching Variational AutoEncodercited by 6
  5. 2025Is There a Better Source Distribution than Gaussian? Exploring Source Distributions for Image Flow Matchingcited by 5
  6. 2026UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Modelscited by 5
  7. 2025MoLingo: Motion-Language Alignment for Text-to-Human Motion Generationcited by 2
  8. 2026Geometry-Aware Image Flow Matchingcited by 2
  9. 2026Understanding Latent Diffusability via Fisher Geometrycited by 0
  10. 2026EditSSC: Toward Editable Semantic Occupancy Scenes with Unconditional Diffusion Modelscited by 0
  11. 2026CLIMB-flow: Coupled Linear Inverse posterior sampling via Multiscale-Based flowcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion Models as Masked Autoencoders
  2. 2024Boosting Latent Diffusion with Perceptual Objectives
  3. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space
  4. 2026LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation
  5. 2026Latent-Compressed Variational Autoencoder for Video Diffusion Models
  6. 2025Diffusion Transformers with Representation Autoencoders
  7. 2023Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models
  8. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learners