mlsanityData to 7 Oct 2026

Diffusion Transformers with Representation Autoencoders

arXiv 2510.11690 ↗First posted 13 Oct 2025cs.CV cs.LGcited by 275

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we explore replacing the VAE with pretrained representation encoders (e.g., DINO, SigLIP, MAE) paired with trained decoders, forming what we term Representation Autoencoders (RAEs). These models provide both high-quality reconstructions and semantically rich latent spaces, while allowing for a scalable transformer-based architecture. Since these latent spaces are typically high-dimensional, a key challenge is enabling diffusion transformers to operate effectively within them. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses. Using a DiT variant equipped with a lightweight, wide DDT head, we achieve strong image generation results on ImageNet: 1.51 FID at 256x256 (no guidance) and 1.13 at both 256x256 and 512x512 (with guidance). RAE offers clear advantages and should be the new default for diffusion transformer training.

Key facts. Cited by 275 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 96 of its 107 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

96 of 107 references.

Closest ancestors · 34

Same direction, newest first

  1. 2025Selective Underfitting in Diffusion Models1 Oct
  2. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space1 Aug
  3. 2025PixNerd: Pixel Neural Field Diffusion31 Jul
  4. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think2 Jul
  5. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesis22 Apr
  6. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers14 Apr
  7. 2025Improving the Diffusability of Autoencoders20 Feb
  8. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Models5 Feb
  9. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  10. 2024Enhancing Sample Generation of Diffusion Models using Noise Level Correction7 Dec
  11. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion25 Oct
  12. 2024Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective16 Oct
22 more
  1. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models14 Oct
  2. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  3. 2024Guiding a Diffusion Model with a Bad Version of Itself4 Jun
  4. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models11 Apr
  5. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer25 Mar
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  7. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  8. 2023Analyzing and Improving the Training Dynamics of Diffusion Models5 Dec
  9. 2023DiffiT: Diffusion Vision Transformers for Image Generation4 Dec
  10. 2023Relay Diffusion: Unifying diffusion process across resolutions for image synthesis4 Sep
  11. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  12. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  13. 2023On the Importance of Noise Scheduling for Diffusion Models26 Jan
  14. 2022Scalable Diffusion Models with Transformers19 Dec
  15. 2022Classifier-Free Diffusion Guidance26 Jul
  16. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  17. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  18. 2021Score-based Generative Modeling in Latent Space10 Jun
  19. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  20. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  21. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  22. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2022Flow Matching for Generative Modelingcited by 5,426
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2016Density estimation using Real NVPcited by 2,029
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  13. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  14. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  15. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  16. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  17. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  18. 2023Vision Transformers Need Registerscited by 844
  19. 2018Image Transformercited by 770
  20. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  21. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  22. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  23. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  24. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  25. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  26. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  27. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  28. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  29. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  30. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
  31. 2025Transfer between Modalities with MetaQueriescited by 231
  32. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  33. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  34. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  35. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  36. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  37. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  38. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 110

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 37; 2026: 238.

275 papers.

Sort:

By area:

  1. 2025Improved Mean Flows: On the Challenges of Fastforward Generative Modelscited by 102
  2. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 75
  3. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  4. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  5. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  6. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  7. 2026Representation Fr'echet Loss for Visual Generationcited by 32
  8. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  9. 2025One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generationcited by 27
  10. 2026Transition Matching Distillation for Fast Video Generationcited by 27
  11. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  12. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
38 more
  1. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 25
  2. 2026Improved Baselines with Representation Autoencoderscited by 24
  3. 2025There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-trainingcited by 23
  4. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  5. 2025The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencodingcited by 21
  6. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 21
  7. 2026Unified Latents (UL): How to train your latentscited by 19
  8. 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 18
  9. 2026One-Step Generative Modeling via Wasserstein Gradient Flowscited by 18
  10. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17
  11. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 17
  12. 2025VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Modelscited by 16
  13. 2025Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusioncited by 16
  14. 2025Adapting Self-Supervised Representations as a Latent Space for Efficient Generationcited by 13
  15. 2025Flow Map Distillation Without Datacited by 13
  16. 2026CoDAR: Continuous Diffusion Language Models are More Powerful Than You Thinkcited by 13
  17. 2025SVG-T2I: Scaling Up Text-to-Image Latent Diffusion Model Without Variational Autoencodercited by 12
  18. 2025Guiding a Diffusion Transformer with the Internal Dynamics of Itselfcited by 12
  19. 2026Learning on the Manifold: Unlocking Standard Diffusion Transformers with Representation Encoderscited by 12
  20. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 12
  21. 2026Repurposing Geometric Foundation Models for Multi-view Diffusioncited by 12
  22. 2026Continuous Latent Diffusion Language Modelcited by 12
  23. 2026Representation Alignment for Just Image Transformers is not Easier than You Thinkcited by 11
  24. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 11
  25. 2025Adversarial Flow Modelscited by 10
  26. 2026PixelGen: Improving Pixel Diffusion with Perceptual Supervisioncited by 10
  27. 2026Representation Distribution Matching for One-Step Visual Generationcited by 10
  28. 2025Continuous Autoregressive Language Modelscited by 9
  29. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 9
  30. 2026RAE-NWM: Navigation World Model in Dense Visual Representation Spacecited by 9
  31. 2026A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokenscited by 9
  32. 2026Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matchingcited by 8
  33. 2026How to Train Your Latent Diffusion Language Model Jointly With the Latent Spacecited by 8
  34. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 8
  35. 2025Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspectivecited by 7
  36. 2025Bidirectional Normalizing Flow: From Data to Noise and Backcited by 7
  37. 2026Hyperspherical Autoencoder for High-Fidelity Image Reconstruction and Generationcited by 7
  38. 2026RPiAE: A Representation-Pivoted Autoencoder Enhancing Both Image Generation and Editingcited by 7

50 most cited of 275 All 275 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Representation Learning with Diffusion Models
  2. 2023Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models
  3. 2026Scaffold Then Internalize: Representation Injection for Diffusion Transformers
  4. 2026The Diffusion Encoder
  5. 2025Exploring Representation-Aligned Latent Space for Better Generation
  6. 2025Geometry-Preserving Encoder/Decoder in Latent Generative Models
  7. 2026Improving Reconstruction of Representation Autoencoder
  8. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representation