mlsanityData to 7 Oct 2026

LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models

arXiv 2405.14477 ↗First posted 23 May 2024cs.LG cs.CVcited by 22

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Advances in latent diffusion models (LDMs) have revolutionized high-resolution image generation, but the design space of the autoencoder that is central to these systems remains underexplored. In this paper, we introduce LiteVAE, a new autoencoder design for LDMs, which leverages the 2D discrete wavelet transform to enhance scalability and computational efficiency over standard variational autoencoders (VAEs) with no sacrifice in output quality. We investigate the training methodologies and the decoder architecture of LiteVAE and propose several enhancements that improve the training dynamics and reconstruction quality. Our base LiteVAE model matches the quality of the established VAEs in current LDMs with a six-fold reduction in encoder parameters, leading to faster training and lower GPU memory requirements, while our larger model outperforms VAEs of comparable complexity across all evaluated metrics (rFID, LPIPS, PSNR, and SSIM).

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 57 of its 76 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

57 of 76 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  2. 2023Analyzing and Improving the Training Dynamics of Diffusion Models5 Dec
  3. 2023CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling26 Oct
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  5. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  6. 2023Efficient Diffusion Training via Min-SNR Weighting Strategy16 Mar
  7. 2023I2SB: Image-to-Image Schr"odinger Bridge12 Feb
  8. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  9. 2022Scalable Diffusion Models with Transformers19 Dec
  10. 2022Wavelet Diffusion Models are fast and scalable Image Generators29 Nov
  11. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidance3 Oct
  12. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications2 Sep
8 more
  1. 2022Wavelet Score-Based Generative Modeling9 Aug
  2. 2022Classifier-Free Diffusion Guidance26 Jul
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  6. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
25 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2018Group Normalizationcited by 1,514
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  8. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,250
  9. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  11. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  12. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  13. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  14. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  15. 2017A General and Adaptive Robust Loss Functioncited by 236
  16. 2018Multi-level Wavelet-CNN for Image Restorationcited by 184
  17. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  18. 2020A U-Net Based Discriminator for Generative Adversarial Networkscited by 128
  19. 2017Real-Time Adaptive Image Compressioncited by 127
  20. 2022Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolutioncited by 114
  21. 2022NAFSSR: Stereo Image Super-Resolution Using NAFNetcited by 75
  22. 2023Designing a Better Asymmetric VQGAN for StableDiffusioncited by 45
  23. 2021WaveFill: A Wavelet-based Generation Network for Image Inpaintingcited by 44
  24. 2022FreGAN: Exploiting Frequency Components for Training GANs under Limited Datacited by 16
  25. 2021Ethics and Creativity in Computer Visioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 4.

22 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2025Improving the Diffusability of Autoencoderscited by 74
  3. 2024WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Modelcited by 55
  4. 2024Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representationcited by 17
  5. 2024Epsilon-VAE: Denoising as Visual Decodingcited by 16
  6. 2025LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Modelscited by 15
  7. 2025Guidance in the Frequency Domain Enables High-Fidelity Sampling at Low CFG Scalescited by 13
  8. 2025H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Modelscited by 12
  9. 2025Latent Diffusion Models with Masked AutoEncoderscited by 11
  10. 2025Decoupling High and Low Frequencies for Faithful Image Generation with Fine Detailscited by 7
  11. 2025SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenizationcited by 7
  12. 2025StyleMe3D: Stylization with Disentangled Priors by Multiple Encoders on 3D Gaussianscited by 6
10 more
  1. 2025Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Modelscited by 5
  2. 2025Latent Harmony: Synergistic Unified UHD Image Restoration via Latent Space Regularization and Controllable Refinementcited by 3
  3. 2025KDC-Diff: A Latent-Aware Diffusion Model with Knowledge Retention for Memory-Efficient Image Generationcited by 0
  4. 2025Latent Posterior-Mean Rectified Flow for Higher-Fidelity Perceptual Face Restorationcited by 0
  5. 2025Discrete Wavelet Transform as a Facilitator for Expressive Latent Space Representation in Variational Autoencoders in Satellite Imagerycited by 0
  6. 2025Wave-GMS: Lightweight Multi-Scale Generative Model for Medical Image Segmentationcited by 0
  7. 2026Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoderscited by 0
  8. 2026Video Generation with Predictive Latentscited by 0
  9. 2026Frequency-Aware Dual-Stream Learning for Balanced Realism and Fidelity in Electron Microscopy Imagingcited by 0
  10. 2026Beyond Compression: Training Latent Representations for Stable Long-Horizon Rollout in Neural Surrogate Solverscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Transformers with Representation Autoencoders
  2. 2026LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation
  3. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space
  4. 2026SRA 2: Variational Autoencoder Self-Representation Alignment for Efficient Diffusion Training
  5. 2026Geometric Autoencoder for Diffusion Models
  6. 2023Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models
  7. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models
  8. 2025One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models