mlsanityData to 14 Oct 2026

Latent Diffusion Model without Variational Autoencoder

arXiv 2510.15301 ↗First posted 17 Oct 2025cs.CV cs.AIcited by 88

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Recent progress in diffusion-based visual generation has largely relied on latent diffusion models with variational autoencoders (VAEs). While effective for high-fidelity synthesis, this VAE+diffusion paradigm suffers from limited training efficiency, slow inference, and poor transferability to broader vision tasks. These issues stem from a key limitation of VAE latent spaces: the lack of clear semantic separation and strong discriminative structure. Our analysis confirms that these properties are crucial not only for perception and understanding tasks, but also for the stable and efficient training of latent diffusion models. Motivated by this insight, we introduce SVG, a novel latent diffusion model without variational autoencoders, which leverages self-supervised representations for visual generation. SVG constructs a feature space with clear semantic discriminability by leveraging frozen DINO features, while a lightweight residual branch captures fine-grained details for high-fidelity reconstruction. Diffusion models are trained directly on this semantically structured latent space to facilitate more efficient learning. As a result, SVG enables accelerated diffusion training, supports few-step sampling, and improves generative quality. Experimental results further show that SVG preserves the semantic and discriminative capabilities of the underlying self-supervised representations, providing a principled pathway toward task-general, high-quality visual representations. Code and interpretations are available at https://howlin-wang.github.io/svg/.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 55 of its 71 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

55 of 71 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think2 Jul
  2. 2025Diffuse and Disperse: Image Generation with Representation Regularization10 Jun
  3. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesis22 Apr
  4. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers14 Apr
  5. 2025CFG-Zero*: Improved Classifier-Free Guidance for Flow Matching Models24 Mar
  6. 2025Improving the Diffusability of Autoencoders20 Feb
  7. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  8. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  10. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  11. 2024PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models10 Jan
  12. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
8 more
  1. 2022Scalable Diffusion Models with Transformers19 Dec
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  3. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representation30 Nov
  4. 2021Score-based Generative Modeling in Latent Space10 Jun
  5. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  7. 2020Denoising Diffusion Implicit Models6 Oct
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
23 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  2. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  5. 2017Improved Training of Wasserstein GANscited by 2,235
  6. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  9. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  12. 2023A Survey on Multimodal Large Language Modelscited by 909
  13. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  14. 2024LTX-Video: Realtime Video Latent Diffusioncited by 581
  15. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  16. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  17. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  18. 2020Query-Key Normalization for Transformerscited by 280
  19. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  20. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  21. 2023Return of Unconditional Generation: A Self-supervised Representation Generation Methodcited by 98
  22. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 89
  23. 2018Spherical Latent Spaces for Stable Variational Autoencoderscited by 67

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 72.

88 papers.

Sort:

By area:

  1. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 83
  2. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 53
  3. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  4. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 32
  5. 2026Improved Baselines with Representation Autoencoderscited by 31
  6. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  7. 2025The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencodingcited by 22
  8. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 22
  9. 2026Unified Latents (UL): How to train your latentscited by 20
  10. 2025Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusioncited by 18
  11. 2025VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Modelscited by 17
  12. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
38 more
  1. 2026Repurposing Geometric Foundation Models for Multi-view Diffusioncited by 15
  2. 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 15
  3. 2026Learning on the Manifold: Unlocking Standard Diffusion Transformers with Representation Encoderscited by 13
  4. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 13
  5. 2026VGGT-World: Transforming VGGT into an Autoregressive Geometry World Modelcited by 13
  6. 2025Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspectivecited by 10
  7. 2026Improving Reconstruction of Representation Autoencodercited by 9
  8. 2026What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusioncited by 9
  9. 2026PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusioncited by 8
  10. 2026RPiAE: A Representation-Pivoted Autoencoder Enhancing Both Image Generation and Editingcited by 7
  11. 2025DINO-Tok: Adapting DINO for Visual Tokenizerscited by 6
  12. 2025Distribution Matching Variational AutoEncodercited by 6
  13. 2025SemanticGen: Video Generation in Semantic Spacecited by 6
  14. 2026DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignmentcited by 6
  15. 2025REGLUE Your Latents with Global and Local Semantics for Entangled Diffusioncited by 5
  16. 2026Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Drivingcited by 5
  17. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  18. 2026End-to-End Autoregressive Image Generation with 1D Semantic Tokenizercited by 5
  19. 2025LaFiTe: A Generative Latent Field for 3D Native Texturingcited by 4
  20. 2025RecTok: Reconstruction Distillation along Rectified Flowcited by 4
  21. 2026FrequencyBooster: Full-Frequency Modeling for High-Fidelity Pixel Diffusioncited by 4
  22. 2026Representation Forcing for Bottleneck-Free Unified Multimodal Modelscited by 4
  23. 2026Flow Matching in Feature Space for Stochastic World Modelingcited by 4
  24. 2026PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Spacecited by 4
  25. 2025Plan-X: Instruct Video Generation via Semantic Planningcited by 3
  26. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  27. 2026Steering Visual Generation in Unified Multimodal Models with Understanding Supervisioncited by 3
  28. 2026WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modelingcited by 3
  29. 2026FREPix: Frequency-Heterogeneous Flow Matching for Pixel-Space Image Generationcited by 3
  30. 2025Latent Diffusion Inversion Requires Understanding the Latent Spacecited by 2
  31. 2025Visual Generation Tuningcited by 2
  32. 2026HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understandingcited by 2
  33. 2026OneWorld: Taming Scene Generation with 3D Unified Representation Autoencodercited by 2
  34. 2026Extending One-Step Image Generation from Class Labels to Text via Discriminative Text Representationcited by 2
  35. 2026RiT: Vanilla Diffusion Transformers Suffice in Representation Spacecited by 2
  36. 2026DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoderscited by 2
  37. 2026IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncodercited by 2
  38. 2026Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusioncited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SVG-T2I: Scaling Up Text-to-Image Latent Diffusion Model Without Variational Autoencoder
  2. 2023Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models
  3. 2024Nested Diffusion Models Using Hierarchical Latent Priors
  4. 2024SVGFusion: A VAE-Diffusion Transformer for Vector Graphic Generation
  5. 2022Understanding Diffusion Models: A Unified Perspective
  6. 2025Diffusion Transformers with Representation Autoencoders
  7. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Models
  8. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents