mlsanityData to 7 Oct 2026

Denoising Diffusion Autoencoders are Unified Self-supervised Learners

arXiv 2303.09769 ↗First posted 17 Mar 2023cs.CV cs.LGcited by 128

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Inspired by recent advances in diffusion models, which are reminiscent of denoising autoencoders, we investigate whether they can acquire discriminative representations for classification via generative pre-training. This paper shows that the networks in diffusion models, namely denoising diffusion autoencoders (DDAE), are unified self-supervised learners: by pre-training on unconditional image generation, DDAE has already learned strongly linear-separable representations within its intermediate layers without auxiliary encoders, thus making diffusion pre-training emerge as a general approach for generative-and-discriminative dual learning. To validate this, we conduct linear probe and fine-tuning evaluations. Our diffusion-based approach achieves 95.9% and 50.0% linear evaluation accuracies on CIFAR-10 and Tiny-ImageNet, respectively, and is comparable to contrastive learning and masked autoencoders for the first time. Transfer learning from ImageNet also confirms the suitability of DDAE for Vision Transformers, suggesting the potential to scale DDAEs as unified foundation models. Code is available at github.com/FutureXiang/ddae.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 57 of its 74 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

57 of 74 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Consistency Models2 Mar
  2. 2022Exploring Vision Transformers as Diffusion Learners28 Dec
  3. 2022Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models26 Dec
  4. 2022Scalable Diffusion Models with Transformers19 Dec
  5. 2022From Points to Functions: Infinite-dimensional Representations in Diffusion Models25 Oct
  6. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  7. 2022Classifier-Free Diffusion Guidance26 Jul
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  9. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  10. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representation30 Nov
  11. 2021Score-Based Generative Classifiers1 Oct
  12. 2021Diffusion Models Beat GANs on Image Synthesis11 May
4 more
  1. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  3. 2020Denoising Diffusion Implicit Models6 Oct
  4. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
29 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Wide Residual Networkscited by 3,057
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  13. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  14. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  16. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  17. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  18. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  19. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  20. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  21. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  22. 2021GAN Inversion: A Surveycited by 366
  23. 2016Adversarial Feature Learningcited by 350
  24. 2020Whitening for Self-Supervised Representation Learningcited by 278
  25. 2019Large Scale Adversarial Representation Learningcited by 238
  26. 2016Adversarially Learned Inferencecited by 216
  27. 2021Diffusion-Based Representation Learningcited by 57
  28. 2021Distill on the Go: Online knowledge distillation in self-supervised learningcited by 22
  29. 2022Exploring the Coordination of Frequency and Attention in Masked Image Modelingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 30; 2025: 55; 2026: 36.

128 papers.

Sort:

By area:

  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  2. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  3. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  4. 2025Mixture of Contexts for Long Video Generationcited by 90
  5. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 89
  6. 2023SODA: Bottleneck Diffusion Models for Representation Learningcited by 76
  7. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  8. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  9. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50
  10. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  11. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
  12. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 42
38 more
  1. 2023Learned representation-guided diffusion models for large-image generationcited by 39
  2. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  3. 2023Do text-free diffusion models learn discriminative visual representations?cited by 33
  4. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perceptioncited by 32
  5. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  6. 2024An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Modelscited by 22
  7. 2025Decoupled MeanFlow: Turning Flow Models into Flow Maps for Accelerated Samplingcited by 19
  8. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modelingcited by 18
  9. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Modelscited by 17
  10. 2023Adversarial Examples Are Not Real Featurescited by 16
  11. 2024Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentationcited by 16
  12. 2024SwiMDiff: Scene-wide Matching Contrastive Learning with Diffusion Constraint for Remote Sensing Imagecited by 14
  13. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 14
  14. 2023Gen2Det: Generate to Detectcited by 12
  15. 2025Learning Diffusion Models with Flexible Representation Guidancecited by 12
  16. 2023Masked Diffusion as Self-supervised Representation Learnercited by 11
  17. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  18. 2025Robust Single-Stage Fully Sparse 3D Object Detection via Detachable Latent Diffusioncited by 11
  19. 2025Can Generative Geospatial Diffusion Models Excel as Discriminative Geospatial Foundation Models?cited by 10
  20. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 10
  21. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
  22. 2024Masked Completion via Structured Diffusion with White-Box Transformerscited by 9
  23. 2024Aligning Diffusion Models with Noise-Conditioned Perceptioncited by 9
  24. 2025Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Modelscited by 9
  25. 2025TriVLA: A Triple-System-Based Unified Vision-Language-Action Model with Episodic World Modeling for General Robot Controlcited by 9
  26. 2024Divot: Diffusion Powers Video Tokenizer for Comprehension and Generationcited by 8
  27. 2025UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Trainingcited by 8
  28. 2025Temporal Score Analysis for Understanding and Correcting Diffusion Artifactscited by 8
  29. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 8
  30. 2024Few-Shot Image Generation by Conditional Relaxing Diffusion Inversioncited by 7
  31. 2024Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generationcited by 7
  32. 2025EgoAgent: A Joint Predictive Agent Model in Egocentric Worldscited by 7
  33. 2025No Alignment Needed for Generation: Learning Linearly Separable Representations in Diffusion Modelscited by 7
  34. 2025LayerSync: Self-aligning Intermediate Layerscited by 7
  35. 2024Contrastive Learning with Synthetic Positivescited by 6
  36. 2025Unleashing the Potential of Pre-Trained Diffusion Models for Generalizable Person Re-Identificationcited by 6
  37. 2025Enhancing Variational Autoencoders with Smooth Robust Latent Encodingcited by 6
  38. 2025Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Modelscited by 6

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion Models as Masked Autoencoders
  2. 2025Automated Learning of Semantic Embedding Representations for Diffusion Models
  3. 2022Representation Learning with Diffusion Models
  4. 2025Latent Diffusion Models with Masked AutoEncoders
  5. 2025Diffusion Transformers with Representation Autoencoders
  6. 2023Conditional Generation from Unconditional Diffusion Models using Denoiser Representations
  7. 2026Should We Skip Diffusion?
  8. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer