mlsanityData to 7 Oct 2026

Diffusion Model as Representation Learner

arXiv 2308.10916 ↗First posted 21 Aug 2023cs.CV cs.AIcited by 95

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion Probabilistic Models (DPMs) have recently demonstrated impressive results on various generative tasks.Despite its promises, the learned representations of pre-trained DPMs, however, have not been fully understood. In this paper, we conduct an in-depth investigation of the representation power of DPMs, and propose a novel knowledge transfer method that leverages the knowledge acquired by generative DPMs for recognition tasks. Our study begins by examining the feature space of DPMs, revealing that DPMs are inherently denoising autoencoders that balance the representation learning with regularizing model capacity. To this end, we introduce a novel knowledge transfer paradigm named RepFusion. Our paradigm extracts representations at different time steps from off-the-shelf DPMs and dynamically employs them as supervision for student networks, in which the optimal time is determined through reinforcement learning. We evaluate our approach on several image classification, semantic segmentation, and landmark detection benchmarks, and demonstrate that it outperforms state-of-the-art methods. Our results uncover the potential of DPMs as a powerful tool for representation learning and provide insights into the usefulness of generative models beyond sample generation. The code is available at urlhttps://github.com/Adamdad/Repfusion.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 64 of its 90 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

64 of 90 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?22 May
  2. 2023Structural Pruning for Diffusion Models18 May
  3. 2022Diffusion Probabilistic Model Made Slim27 Nov
  4. 2022From Points to Functions: Infinite-dimensional Representations in Diffusion Models25 Oct
  5. 2022Diffusion Models already have a Semantic Latent Space20 Oct
  6. 2022On Distillation of Guided Diffusion Models6 Oct
  7. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representation30 Nov
  8. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  10. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
38 more
  1. 2016Wide Residual Networkscited by 3,057
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2022Video Diffusion Modelscited by 2,280
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  10. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  11. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  12. 2019Relational Knowledge Distillationcited by 873
  13. 2019Contrastive Representation Distillationcited by 734
  14. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  15. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  16. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  17. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  18. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  19. 2023Emergent Correspondence from Image Diffusioncited by 437
  20. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  21. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  22. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  23. 2017Data Augmentation Generative Adversarial Networkscited by 278
  24. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  25. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  26. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  27. 2023DepGraph: Towards Any Structural Pruningcited by 202
  28. 2018Look at Boundary: A Boundary-Aware Face Alignment Algorithmcited by 137
  29. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  30. 2022Dataset Distillation via Factorizationcited by 115
  31. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113
  32. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  33. 2022Deep Model Reassemblycited by 78
  34. 2022Factorizing Knowledge in Neural Networkscited by 59
  35. 2021Diffusion-Based Representation Learningcited by 57
  36. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  37. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Modelscited by 37
  38. 2022Learning Graph Neural Networks for Image Style Transfercited by 29

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 29; 2025: 35; 2026: 29.

95 papers.

Sort:

By area:

  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  2. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  3. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  4. 2024Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Imagescited by 69
  5. 2023Priority-Centric Human Motion Generation in Discrete Latent Spacecited by 66
  6. 2024MindBridge: A Cross-Subject Brain Decoding Frameworkcited by 65
  7. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  8. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50
  9. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
  10. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 42
  11. 2026Image Generators are Generalist Vision Learnerscited by 40
  12. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion modelscited by 29
38 more
  1. 2024Video Diffusion Models: A Surveycited by 28
  2. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  3. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Featurescited by 23
  4. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  5. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  6. 2024PartSTAD: 2D-to-3D Part Segmentation Task Adaptationcited by 18
  7. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modelingcited by 18
  8. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  9. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 14
  10. 2025On Denoising Walking Videos for Gait Recognitioncited by 14
  11. 2024Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Modelscited by 12
  12. 2024Visual Lexicon: Rich Image Features in Language Spacecited by 12
  13. 2025VDT-Auto: End-to-end Autonomous Driving with VLM-Guided Diffusion Transformerscited by 12
  14. 2025Learning Diffusion Models with Flexible Representation Guidancecited by 12
  15. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  16. 2024Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Controlcited by 11
  17. 2025SARA: Structural and Adversarial Representation Alignment for Training-efficient Diffusion Modelscited by 11
  18. 2024Residual Connections Harm Generative Representation Learningcited by 10
  19. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 10
  20. 2023Guided Diffusion from Self-Supervised Diffusion Featurescited by 9
  21. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 8
  22. 2024RetinaRegNet: A Zero-Shot Approach for Retinal Image Registrationcited by 7
  23. 2024Contrastive Learning with Synthetic Positivescited by 6
  24. 2024FiffDepth: Feed-forward Transformation of Diffusion-Based Generators for Detailed Depth Estimationcited by 5
  25. 2025Diffusion Models for Computational Neuroimaging: A Surveycited by 5
  26. 2025Efficient Generative Model Training via Embedded Representation Warmupcited by 5
  27. 2025DiffNMR: Diffusion Models for Nuclear Magnetic Resonance Spectra Elucidationcited by 5
  28. 2025VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Drivingcited by 5
  29. 2026BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamicscited by 5
  30. 2025Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Modelscited by 4
  31. 2026DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformerscited by 4
  32. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Modelscited by 3
  33. 2024Distilled Datamodel with Reverse Gradient Matchingcited by 3
  34. 2024Denoising Diffusion Models for Anomaly Localization in Medical Imagescited by 3
  35. 2025PosDiffAE: Position-aware Diffusion Auto-encoder For High-Resolution Brain Tissue Classification Incorporating Artifact Restorationcited by 3
  36. 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learningcited by 3
  37. 2025High-Performance Self-Supervised Learning by Joint Training of Flow Matchingcited by 3
  38. 2026GenMask: Adapting DiT for Segmentation via Direct Mask Generationcited by 3

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspect
  2. 2022Representation Learning with Diffusion Models
  3. 2022Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models
  4. 2024An overview of diffusion models for generative artificial intelligence
  5. 2026Representation-Conditioned Diffusion Models for Guided Training Data Generation
  6. 2026Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector
  7. 2023A Comprehensive Survey on Knowledge Distillation of Diffusion Models
  8. 2023Improving Denoising Diffusion Probabilistic Models via Exploiting Shared Representations