mlsanityData to 14 Oct 2026

Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model

arXiv 2208.07791 ↗First posted 16 Aug 2022cs.CVcited by 58

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion Denoising Probability Models (DDPM) and Vision Transformer (ViT) have demonstrated significant progress in generative tasks and discriminative tasks, respectively, and thus far these models have largely been developed in their own domains. In this paper, we establish a direct connection between DDPM and ViT by integrating the ViT architecture into DDPM, and introduce a new generative model called Generative ViT (GenViT). The modeling flexibility of ViT enables us to further extend GenViT to hybrid discriminative-generative modeling, and introduce a Hybrid ViT (HybViT). Our work is among the first to explore a single ViT for image generation and classification jointly. We conduct a series of experiments to analyze the performance of proposed models and demonstrate their superiority over prior state-of-the-arts in both generative and discriminative tasks. Our code and pre-trained models can be found in https://github.com/sndnyang/Diffusion_ViT .

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 57 of its 68 references that are in the map. First posted on arXiv 16 Aug 2022.

Builds on

57 of 68 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models17 Jan
  2. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  3. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  12. 2013Intriguing properties of neural networkscited by 5,155
38 more
  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  6. 2016Improved Techniques for Training GANscited by 2,970
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  14. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  15. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  16. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  17. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  18. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
  19. 2014Semi-Supervised Learning with Deep Generative Modelscited by 496
  20. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  21. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  22. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  23. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  24. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  25. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  26. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  27. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 130
  28. 2020A U-Net Based Discriminator for Generative Adversarial Networkscited by 128
  29. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 123
  30. 2021Vision Transformer for Small-Size Datasetscited by 122
  31. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  32. 2020Learning Latent Space Energy-Based Prior Modelcited by 104
  33. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 104
  34. 2020Generalized Energy Based Modelscited by 73
  35. 2020No MCMC for me: Amortized sampling for fast and stable training of energy-based modelscited by 64
  36. 2017Triple Generative Adversarial Netscited by 60
  37. 2019Hybrid Models with Deep and Invertible Featurescited by 41
  38. 2021JEM++: Improved Techniques for Training JEMcited by 28

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 20; 2024: 20; 2025: 9; 2026: 5.

58 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  2. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  3. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  4. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 262
  5. 2023Fast Training of Diffusion Models with Masked Transformerscited by 176
  6. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 142
  7. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 130
  8. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 118
  9. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 109
  10. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 106
  11. 2023Robust Classification via a Single Diffusion Modelcited by 79
  12. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 79
38 more
  1. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  2. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  3. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attentioncited by 56
  4. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  5. 2022Efficient Diffusion Models for Vision: A Surveycited by 51
  6. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 49
  7. 2023Class-Balancing Diffusion Modelscited by 48
  8. 2024Scalable Diffusion Models with State Space Backbonecited by 45
  9. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  10. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 37
  11. 2023Latent Space Editing in Transformer-Based Flow Matchingcited by 32
  12. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Expertscited by 31
  13. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  14. 2023Denoising Task Routing for Diffusion Modelscited by 28
  15. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  16. 2023ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connectioncited by 25
  17. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  18. 2023Learning Data Representations with Joint Diffusion Modelscited by 22
  19. 2023DiffuVolume: Diffusion Model for Volume based Stereo Matchingcited by 16
  20. 2023Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Modelscited by 13
  21. 2023Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modelingcited by 13
  22. 2024A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Modelscited by 13
  23. 2025Learning Diffusion Models with Flexible Representation Guidancecited by 13
  24. 2025SARA: Structural and Adversarial Representation Alignment for Training-efficient Diffusion Modelscited by 11
  25. 2022Exploring Vision Transformers as Diffusion Learnerscited by 10
  26. 2023Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Surveycited by 10
  27. 2023Masked Diffusion Models Are Fast Distribution Learnerscited by 7
  28. 2025Unleashing the Potential of Pre-Trained Diffusion Models for Generalizable Person Re-Identificationcited by 7
  29. 2023Dual Diffusion Architecture for Fisheye Image Rectification: Synthetic-to-Real Generalizationcited by 5
  30. 2023Towards understanding neural collapse in supervised contrastive learning with the information bottleneck methodcited by 5
  31. 2024Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoisingcited by 5
  32. 2025Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentialscited by 4
  33. 2023DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capabilitycited by 3
  34. 2026GenMask: Adapting DiT for Segmentation via Direct Mask Generationcited by 3
  35. 2026Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representationscited by 3
  36. 2024Pretrained Reversible Generation as Unsupervised Visual Representation Learningcited by 2
  37. 2025Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptationcited by 2
  38. 2025Beyond the Generative Learning Trilemma: Generative Model Assessment in Data Scarcity Domainscited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Few-Shot Diffusion Models
  2. 2024Diffusion Models in Low-Level Vision: A Survey
  3. 2024Generative AI in Vision: A Survey on Models, Metrics and Applications
  4. 2022Diffusion Models in Vision: A Survey
  5. 2023Improving Denoising Diffusion Probabilistic Models via Exploiting Shared Representations
  6. 2024DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space
  7. 2023Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback
  8. 2022Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance