mlsanityData to 7 Oct 2026

Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models

arXiv 2501.01423 ↗First posted 2 Jan 2025cs.CV cs.LGcited by 376

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Latent diffusion models with Transformer architectures excel at generating high-fidelity images. However, recent studies reveal an optimization dilemma in this two-stage design: while increasing the per-token feature dimension in visual tokenizers improves reconstruction quality, it requires substantially larger diffusion models and more training iterations to achieve comparable generation performance. Consequently, existing systems often settle for sub-optimal solutions, either producing visual artifacts due to information loss within tokenizers or failing to converge fully due to expensive computation costs. We argue that this dilemma stems from the inherent difficulty in learning unconstrained high-dimensional latent spaces. To address this, we propose aligning the latent space with pre-trained vision foundation models when training the visual tokenizers. Our proposed VA-VAE (Vision foundation model Aligned Variational AutoEncoder) significantly expands the reconstruction-generation frontier of latent diffusion models, enabling faster convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces. To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with improved training strategies and architecture designs, termed LightningDiT. The integrated system achieves state-of-the-art (SOTA) performance on ImageNet 256x256 generation with an FID score of 1.35 while demonstrating remarkable training efficiency by reaching an FID score of 2.11 in just 64 epochs--representing an over 21 times convergence speedup compared to the original DiT. Models and codes are available at: https://github.com/hustvl/LightningDiT.

Key facts. Cited by 376 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 42 of its 49 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

42 of 49 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models14 Oct
  2. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers14 Oct
  3. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification14 Oct
  4. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  5. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers9 May
  6. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models11 Apr
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  8. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  10. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  11. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  12. 2022Scalable Diffusion Models with Transformers19 Dec
1 more
  1. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec

Other references · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023Segment Anythingcited by 8,158
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
17 more
  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  3. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  4. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  5. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  6. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  7. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  8. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  9. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  10. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  11. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  12. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  13. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  14. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attentioncited by 56
  15. 2023Intriguing Properties of Quantization at Scalecited by 48
  16. 2022Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesiscited by 31
  17. 2024Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Predictioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 132; 2026: 244.

376 papers.

Sort:

By area:

  1. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  2. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 212
  3. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  4. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  5. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  6. 2025Improved Mean Flows: On the Challenges of Fastforward Generative Modelscited by 102
  7. 2025DDT: Decoupled Diffusion Transformercited by 100
  8. 2026Generative Modeling via Driftingcited by 98
  9. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  10. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  11. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  12. 2025Latent Diffusion Model without Variational Autoencodercited by 83
38 more
  1. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 76
  2. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
  3. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 75
  4. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  5. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
  6. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 55
  7. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 55
  8. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  9. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50
  10. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  11. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  12. 2025Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformerscited by 47
  13. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Spacecited by 44
  14. 2025DiP: Taming Diffusion Models in Pixel Spacecited by 42
  15. 2025Efficient Diffusion Models: A Surveycited by 35
  16. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  17. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  18. 2025REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Trainingcited by 33
  19. 2026Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generationcited by 33
  20. 2026Representation Fr'echet Loss for Visual Generationcited by 32
  21. 2025FlowTok: Flowing Seamlessly Across Text and Image Tokenscited by 31
  22. 2025AToken: A Unified Tokenizer for Visioncited by 30
  23. 2025Bridging Continuous and Discrete Tokens for Autoregressive Visual Generationcited by 29
  24. 2025U-REPA: Aligning Diffusion U-Nets to ViTscited by 29
  25. 2025Diffusion Models without Classifier-free Guidancecited by 28
  26. 2026Qwen-Image-2.0 Technical Reportcited by 28
  27. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 27
  28. 2025One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generationcited by 27
  29. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  30. 2025"Principal Components" Enable A New Language of Imagescited by 25
  31. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 25
  32. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  33. 2026Improved Baselines with Representation Autoencoderscited by 24
  34. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  35. 2025There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-trainingcited by 23
  36. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 22
  37. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  38. 2025TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Trainingcited by 21

50 most cited of 376 All 376 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
  2. 2026Latent-Compressed Variational Autoencoder for Video Diffusion Models
  3. 2025Exploring Representation-Aligned Latent Space for Better Generation
  4. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesis
  5. 2025Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization
  6. 2022Understanding Diffusion Models: A Unified Perspective
  7. 2026Diffusing in the Right Space: A Systematic Study of Latent Diffusability
  8. 2024Nested Diffusion Models Using Hierarchical Latent Priors