# Dream the Impossible: Outlier Imagination with Diffusion Models

arXiv: https://arxiv.org/abs/2309.13415 · HTML: https://mlsanity.com/paper/2309.13415/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 111 of its 134 references that are in the map.
- First posted on arXiv 23 Sep 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 111)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)

## Built on by (most cited first, 30 of 77)

- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [Process Reward Model with Q-Value Rankings](https://mlsanity.com/paper/2410.11287/) (2024)
- [How Does Unlabeled Data Provably Help Out-of-Distribution Detection?](https://mlsanity.com/paper/2402.03502/) (2024)
- [Noise-Consistent Siamese-Diffusion for Medical Image Synthesis and Segmentation](https://mlsanity.com/paper/2505.06068/) (2025)
- [A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2311.03524/) (2023)
- [ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object](https://mlsanity.com/paper/2403.18775/) (2024)
- [Self-Guided Generation of Minority Samples Using Diffusion Models](https://mlsanity.com/paper/2407.11555/) (2024)
- [TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Sampling](https://mlsanity.com/paper/2310.01819/) (2023)
- [GOLD: Graph Out-of-Distribution Detection via Implicit Adversarial Latent Generation](https://mlsanity.com/paper/2502.05780/) (2025)
- [Placing Objects in Context via Inpainting for Out-of-distribution Segmentation](https://mlsanity.com/paper/2402.16392/) (2024)
- [Can OOD Object Detectors Learn from Foundation Models?](https://mlsanity.com/paper/2409.05162/) (2024)
- [FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector](https://mlsanity.com/paper/2412.05293/) (2024)
- [Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2410.02681/) (2024)
- [On the Learnability of Out-of-distribution Detection](https://mlsanity.com/paper/2404.04865/) (2024)
- [PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomalies](https://mlsanity.com/paper/2506.09237/) (2025)
- [GDSG: Graph Diffusion-based Solution Generator for Optimization Problems in MEC Networks](https://mlsanity.com/paper/2412.08296/) (2024)
- [RODEO: Robust Outlier Detection via Exposing Adaptive Out-of-Distribution Samples](https://mlsanity.com/paper/2501.16971/) (2025)
- [AdaSCALE: Adaptive Scaling for OOD Detection](https://mlsanity.com/paper/2503.08023/) (2025)
- [When and How Does In-Distribution Label Help Out-of-Distribution Detection?](https://mlsanity.com/paper/2405.18635/) (2024)
- [Out-of-Distribution Learning with Human Feedback](https://mlsanity.com/paper/2408.07772/) (2024)
- [Adversarially Robust Out-of-Distribution Detection Using Lyapunov-Stabilized Embeddings](https://mlsanity.com/paper/2410.10744/) (2024)
- [DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automaton](https://mlsanity.com/paper/2402.04411/) (2024)
- [Boost-and-Skip: A Simple Guidance-Free Diffusion for Minority Generation](https://mlsanity.com/paper/2502.06516/) (2025)
- [Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognition](https://mlsanity.com/paper/2508.08004/) (2025)
- [Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detection](https://mlsanity.com/paper/2510.10584/) (2025)
- [GOOD: Training-Free Guided Diffusion Sampling for Out-of-Distribution Detection](https://mlsanity.com/paper/2510.17131/) (2025)
- [Future-Proofing Class-Incremental Learning](https://mlsanity.com/paper/2404.03200/) (2024)
- [Robust Disaster Assessment from Aerial Imagery Using Text-to-Image Synthetic Data](https://mlsanity.com/paper/2405.13779/) (2024)
- [Diffusion based Semantic Outlier Generation via Nuisance Awareness for Out-of-Distribution Detection](https://mlsanity.com/paper/2408.14841/) (2024)
- [Proto-OOD: Enhancing OOD Object Detection with Prototype Feature Similarity](https://mlsanity.com/paper/2409.05466/) (2024)

## Abstract (arXiv)

Utilizing auxiliary outlier datasets to regularize the machine learning model has demonstrated promise for out-of-distribution (OOD) detection and safe prediction. Due to the labor intensity in data collection and cleaning, automating outlier data generation has been a long-desired alternative. Despite the appeal, generating photo-realistic outliers in the high dimensional pixel space has been an open challenge for the field. To tackle the problem, this paper proposes a new framework DREAM-OOD, which enables imagining photo-realistic outliers by way of diffusion models, provided with only the in-distribution (ID) data and classes. Specifically, DREAM-OOD learns a text-conditioned latent space based on ID data, and then samples outliers in the low-likelihood region via the latent, which can be decoded into images by the diffusion model. Different from prior works, DREAM-OOD enables visualizing and understanding the imagined outliers, directly in the pixel space. We conduct comprehensive quantitative and qualitative studies to understand the efficacy of DREAM-OOD, and show that training with the samples generated by DREAM-OOD can benefit OOD detection performance. Code is publicly available at https://github.com/deeplearning-wisc/dream-ood.