mlsanityData to 14 Oct 2026

Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think

arXiv 2410.06940 ↗First posted 9 Oct 2024cs.CV cs.LGcited by 703

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Recent studies have shown that the denoising process in (generative) diffusion models can induce meaningful (discriminative) representations inside the model, though the quality of these representations still lags behind those learned through recent self-supervised learning methods. We argue that one main bottleneck in training large-scale diffusion models for generation lies in effectively learning these representations. Moreover, training can be made easier by incorporating high-quality external visual representations, rather than relying solely on the diffusion models to learn them independently. We study this by introducing a straightforward regularization called REPresentation Alignment (REPA), which aligns the projections of noisy input hidden states in denoising networks with clean image representations obtained from external, pretrained visual encoders. The results are striking: our simple strategy yields significant improvements in both training efficiency and generation quality when applied to popular diffusion and flow-based transformers, such as DiTs and SiTs. For instance, our method can speed up SiT training by over 17.5×, matching the performance (without classifier-free guidance) of a SiT-XL model trained for 7M steps in less than 400K steps. In terms of final generation quality, our approach achieves state-of-the-art results of FID=1.42 using classifier-free guidance with the guidance interval.

Key facts. Cited by 703 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 85 of its 97 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

85 of 97 references.

Closest ancestors · 37

Same direction, newest first

  1. 2024Diffusion Models and Representation Learning: A Survey30 Jun
  2. 2024Distilling Diffusion Models into Conditional GANs9 May
  3. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models11 Apr
  4. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer25 Mar
  5. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation18 Mar
  6. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation7 Mar
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  8. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learning25 Jan
  9. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  10. 2023Analyzing and Improving the Training Dynamics of Diffusion Models5 Dec
  11. 2023DiffiT: Diffusion Vision Transformers for Image Generation4 Dec
  12. 2023SODA: Bottleneck Diffusion Models for Representation Learning29 Nov
25 more
  1. 2023Adversarial Diffusion Distillation28 Nov
  2. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis30 Sep
  3. 2023Diffusion Model as Representation Learner21 Aug
  4. 2023Diffusion Models Beat GANs on Image Classification17 Jul
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul
  6. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  7. 2023ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Process8 Jun
  8. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  9. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learners17 Mar
  10. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation1 Mar
  11. 2023Learning Data Representations with Joint Diffusion Models31 Jan
  12. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  13. 2022Scalable Diffusion Models with Transformers19 Dec
  14. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  15. 2022Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis29 Aug
  16. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  17. 2022Classifier-Free Diffusion Guidance26 Jul
  18. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  19. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  20. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesis29 Nov
  21. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  22. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  23. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  24. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  25. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2022Flow Matching for Generative Modelingcited by 5,702
36 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2021ViViT: A Video Vision Transformercited by 1,620
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  11. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  12. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  13. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  14. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  15. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  16. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  17. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  18. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  19. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  20. 2024The Platonic Representation Hypothesiscited by 382
  21. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  22. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  23. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  24. 2021Generating Images with Sparse Representationscited by 266
  25. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  26. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  27. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  28. 2021Projected GANs Converge Fastercited by 208
  29. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 162
  30. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  31. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 103
  32. 2023Return of Unconditional Generation: A Self-supervised Representation Generation Methodcited by 98
  33. 2021Ensembling Off-the-shelf Models for GAN Trainingcited by 86
  34. 2021Diffusion-Based Representation Learningcited by 57
  35. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 42
  36. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Modelscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 257; 2026: 427.

703 papers.

Sort:

By area:

  1. 2025Mean Flows for One-step Generative Modelingcited by 561
  2. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  4. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  5. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 225
  6. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  7. 2025Seedream 3.0 Technical Reportcited by 154
  8. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  9. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  10. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 132
  11. 2026Generative Modeling via Driftingcited by 116
  12. 2025FLARE: Robot Learning with Implicit World Modelingcited by 113
38 more
  1. 2025Improved Mean Flows: On the Challenges of Fastforward Generative Modelscited by 110
  2. 2025DDT: Decoupled Diffusion Transformercited by 109
  3. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 109
  4. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 102
  5. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 98
  6. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 96
  7. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
  8. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 89
  9. 2025Latent Diffusion Model without Variational Autoencodercited by 88
  10. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  11. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 84
  12. 2025PixelDiT: Pixel Diffusion Transformers for Image Generationcited by 83
  13. 2025PixNerd: Pixel Neural Field Diffusioncited by 82
  14. 2025Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generationcited by 71
  15. 2025Diffuse and Disperse: Image Generation with Representation Regularizationcited by 70
  16. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  17. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 60
  18. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 60
  19. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 57
  20. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 56
  21. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 56
  22. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 56
  23. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 53
  24. 2026One-step Latent-free Image Generation with Pixel Mean Flowscited by 53
  25. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  26. 2025Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformerscited by 51
  27. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 50
  28. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  29. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Spacecited by 47
  30. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  31. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 44
  32. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  33. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 41
  34. 2025AlphaFlow: Understanding and Improving MeanFlow Modelscited by 41
  35. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 38
  36. 2025REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Trainingcited by 37
  37. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  38. 2026Representation Fr'echet Loss for Visual Generationcited by 37

50 most cited of 703 All 703 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Purify Before You Align: Improving Representation Alignment for Diffusion Models
  2. 2022Representation Learning with Diffusion Models
  3. 2022Image Embedding for Denoising Generative Models
  4. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspect
  5. 2026Representation-Conditioned Diffusion Models for Guided Training Data Generation
  6. 2023Diffusion Models as Masked Autoencoders
  7. 2023Conditional Generation from Unconditional Diffusion Models using Denoiser Representations
  8. 2023Masked Diffusion Models Are Fast Distribution Learners