# Understanding the Limitations of Conditional Generative Models

arXiv: https://arxiv.org/abs/1906.01171 · HTML: https://mlsanity.com/paper/1906.01171/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 28 of 28)

- [Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples](https://mlsanity.com/paper/2302.04578/) (2023)
- [CARD: Classification and Regression Diffusion Models](https://mlsanity.com/paper/2206.07275/) (2022)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Score-Based Generative Classifiers](https://mlsanity.com/paper/2110.00473/) (2021)
- [Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence](https://mlsanity.com/paper/2412.18164/) (2024)
- [Invertible DenseNets with Concatenated LipSwish](https://mlsanity.com/paper/2102.02694/) (2021)
- [Evaluating State-of-the-Art Classification Models Against Bayes Optimality](https://mlsanity.com/paper/2106.03357/) (2021)
- [Visual Affordance Prediction for Guiding Robot Exploration](https://mlsanity.com/paper/2305.17783/) (2023)
- [Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better](https://mlsanity.com/paper/2402.00263/) (2024)
- [Understanding Out-of-distribution: A Perspective of Data Dynamics](https://mlsanity.com/paper/2111.14730/) (2021)
- [Topology-Matching Normalizing Flows for Out-of-Distribution Detection in Robot Learning](https://mlsanity.com/paper/2311.06481/) (2023)
- [SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition](https://mlsanity.com/paper/2202.04849/) (2022)
- [Gradient-based inference of abstract task representations for generalization in neural networks](https://mlsanity.com/paper/2407.17356/) (2024)
- [This Probably Looks Exactly Like That: An Invertible Prototypical Network](https://mlsanity.com/paper/2407.12200/) (2024)
- [Probing the Effect of Selection Bias on Generalization: A Thought Experiment](https://mlsanity.com/paper/2105.09934/) (2021)
- [Generative vs. Discriminative modeling under the lens of uncertainty quantification](https://mlsanity.com/paper/2406.09172/) (2024)
- [Learning conditional distributions on continuous spaces](https://mlsanity.com/paper/2406.09375/) (2024)
- [Pretrained Reversible Generation as Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2412.01787/) (2024)
- [ECGTwin: Personalized ECG Generation Using Controllable Diffusion Model](https://mlsanity.com/paper/2508.02720/) (2025)
- [Your VAR Model is Secretly an Efficient and Explainable Generative Classifier](https://mlsanity.com/paper/2510.12060/) (2025)
- [Time-reversed Flow Matching with Worst Transport in High-dimensional Latent Space for Image Anomaly Detection](https://mlsanity.com/paper/2508.05461/) (2025)
- [Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck](https://mlsanity.com/paper/2303.14096/) (2023)
- [Training Image Derivatives: Increased Accuracy and Universal Robustness](https://mlsanity.com/paper/2310.14045/) (2023)
- [Intriguing Properties of Modern GANs](https://mlsanity.com/paper/2402.14098/) (2024)
- [Causal Interpretability for Adversarial Robustness: A Hybrid Generative Classification Approach](https://mlsanity.com/paper/2412.20025/) (2024)
- [ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation](https://mlsanity.com/paper/2505.20498/) (2025)
- [Emotion Diffusion Classifier with Adaptive Margin Discrepancy Training for Facial Expression Recognition](https://mlsanity.com/paper/2603.29578/) (2026)
- [ARMOR: Manifold-Oriented Training for Adversarially Robust Aerial Object Detection under Data Scarcity](https://mlsanity.com/paper/2608.29510/) (2026)

## Abstract (arXiv)

Class-conditional generative models hold promise to overcome the shortcomings of their discriminative counterparts. They are a natural choice to solve discriminative tasks in a robust manner as they jointly optimize for predictive performance and accurate modeling of the input distribution. In this work, we investigate robust classification with likelihood-based generative models from a theoretical and practical perspective to investigate if they can deliver on their promises. Our analysis focuses on a spectrum of robustness properties: (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled in-distribution inputs. Our theoretical result reveals that it is impossible to guarantee detectability of adversarially-perturbed inputs even for near-optimal generative classifiers. Experimentally, we find that while we are able to train robust models for MNIST, robustness completely breaks down on CIFAR10. We relate this failure to various undesirable model properties that can be traced to the maximum likelihood training objective. Despite being a common choice in the literature, our results indicate that likelihood-based conditional generative models may are surprisingly ineffective for robust classification.