mlsanityData to 7 Oct 2026

Learning Data Representations with Joint Diffusion Models

arXiv 2301.13622 ↗First posted 31 Jan 2023cs.LG cs.CV stat.MLcited by 22

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Joint machine learning models that allow synthesizing and classifying data often offer uneven performance between those tasks or are unstable to train. In this work, we depart from a set of empirical observations that indicate the usefulness of internal representations built by contemporary deep diffusion-based generative models not only for generating but also predicting. We then propose to extend the vanilla diffusion model with a classifier that allows for stable joint end-to-end training with shared parameterization between those objectives. The resulting joint diffusion model outperforms recent state-of-the-art hybrid methods in terms of both classification and generation quality on all evaluated benchmarks. On top of our joint training approach, we present how we can directly benefit from shared generative and discriminative representations by introducing a method for visual counterfactual explanations.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 35 of its 51 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

35 of 51 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  2. 2022Classifier-Free Diffusion Guidance26 Jul
  3. 2021Variational Diffusion Models1 Jul
  4. 2021Score-based Generative Modeling in Latent Space10 Jun
  5. 2021A Variational Perspective on Diffusion-Based Generative Models and Score Matching5 Jun
  6. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  7. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  9. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 26

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  5. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  6. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  7. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  8. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  9. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  10. 2018Assessing Generative Models via Precision and Recallcited by 401
  11. 2019Residual Flows for Invertible Generative Modelingcited by 256
  12. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 168
14 more
  1. 2018A Variational U-Net for Conditional Appearance and Shape Generationcited by 124
  2. 2019DIVA: Domain Invariant Variational Autoencoderscited by 114
  3. 2022Diffusion Visual Counterfactual Explanationscited by 96
  4. 2020No MCMC for me: Amortized sampling for fast and stable training of energy-based modelscited by 64
  5. 2021Diffusion-Based Representation Learningcited by 57
  6. 2019Hybrid Models with Deep and Invertible Featurescited by 41
  7. 2021JEM++: Improved Techniques for Training JEMcited by 28
  8. 2017Wasserstein Introspective Neural Networkscited by 18
  9. 2021Invertible DenseNets with Concatenated LipSwishcited by 17
  10. 2022Chroma-VAE: Mitigating Shortcut Learning with Generative Classifierscited by 17
  11. 2014Scheduled denoising autoencoderscited by 11
  12. 2023A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEscited by 9
  13. 2021On robustness of generative representations against catastrophic forgettingcited by 5
  14. 2017Hybrid VAE: Improving Deep Generative Models using Partial Observationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 10; 2026: 4.

22 papers.

Sort:

By area:

  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  2. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  3. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  4. 2023CrowdDiff: Multi-hypothesis Crowd Density Estimation using Diffusion Modelscited by 20
  5. 2025Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modelingcited by 18
  6. 2025SARA: Structural and Adversarial Representation Alignment for Training-efficient Diffusion Modelscited by 11
  7. 2024Joint Diffusion models in Continual Learningcited by 6
  8. 2025Unleashing the Potential of Pre-Trained Diffusion Models for Generalizable Person Re-Identificationcited by 6
  9. 2023Robust semi-supervised segmentation with timestep ensembling diffusion modelscited by 4
  10. 2025EEGDM: EEG Representation Learning via Generative Diffusion Modelcited by 4
  11. 2024Explaining latent representations of generative models with large multimodal modelscited by 3
  12. 2024EvolvED: Evolutionary Embeddings to Understand the Generation Process of Diffusion Modelscited by 3
10 more
  1. 2025V-CECE: Visual Counterfactual Explanations via Conceptual Editscited by 3
  2. 2026BindCLIP: A Unified Contrastive-Generative Representation Learning Framework for Virtual Screeningcited by 2
  3. 2026Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representationscited by 2
  4. 2025Beyond Batch Learning: Global Awareness Enhanced Domain Adaptationcited by 1
  5. 2025Noise Optimized Conditional Diffusion for Domain Adaptationcited by 1
  6. 2025CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Drivingcited by 1
  7. 2025Scalable Energy-Based Models via Adversarial Training: Unifying Discrimination and Generationcited by 1
  8. 2026Evaluating the Representation Space of Diffusion Models via Self-Supervised Principlescited by 1
  9. 2025Global Variational Inference Enhanced Robust Domain Adaptationcited by 0
  10. 2026U-CECE: A Universal Multi-Resolution Framework for Conceptual Counterfactual Explanationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Do text-free diffusion models learn discriminative visual representations?
  2. 2023Diffusion Models Beat GANs on Image Classification
  3. 2025Generalization of Diffusion Models Arises with a Balanced Representation Space
  4. 2024Diffusion Models For Multi-Modal Generative Modeling
  5. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspect
  6. 2025Learning Diffusion Models with Flexible Representation Guidance
  7. 2026Representation-Conditioned Diffusion Models for Guided Training Data Generation
  8. 2022Representation Learning with Diffusion Models