# Diffusion Models as Data Mining Tools

arXiv: https://arxiv.org/abs/2408.02752 · HTML: https://mlsanity.com/paper/2408.02752/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 20 Jul 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence](https://mlsanity.com/paper/2305.14334/) (2023)
- [Generative Models as a Data Source for Multiview Representation Learning](https://mlsanity.com/paper/2106.05258/) (2021)
- [A Century of Portraits: A Visual Historical Record of American High School Yearbooks](https://mlsanity.com/paper/1511.02575/) (2015)
- [Ensembling with Deep Generative Views](https://mlsanity.com/paper/2104.14551/) (2021)
- [Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning](https://mlsanity.com/paper/1903.02678/) (2019)
- [StreetStyle: Exploring world-wide clothing styles from millions of photos](https://mlsanity.com/paper/1706.01869/) (2017)
- [Learning Co-segmentation by Segment Swapping for Retrieval and Discovery](https://mlsanity.com/paper/2110.15904/) (2021)

## Built on by (most cited first, 5 of 5)

- [Learning a Generative Meta-Model of LLM Activations](https://mlsanity.com/paper/2602.06964/) (2026)
- [GPS as a Control Signal for Image Generation](https://mlsanity.com/paper/2501.12390/) (2025)
- [Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selection](https://mlsanity.com/paper/2412.09959/) (2024)
- [Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI](https://mlsanity.com/paper/2411.15265/) (2024)
- [Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images](https://mlsanity.com/paper/2504.08727/) (2025)

## Abstract (arXiv)

This paper demonstrates how to use generative models trained for image synthesis as tools for visual data mining. Our insight is that since contemporary generative models learn an accurate representation of their training data, we can use them to summarize the data by mining for visual patterns. Concretely, we show that after finetuning conditional diffusion models to synthesize images from a specific dataset, we can use these models to define a typicality measure on that dataset. This measure assesses how typical visual elements are for different data labels, such as geographic location, time stamps, semantic labels, or even the presence of a disease. This analysis-by-synthesis approach to data mining has two key advantages. First, it scales much better than traditional correspondence-based approaches since it does not require explicitly comparing all pairs of visual elements. Second, while most previous works on visual data mining focus on a single dataset, our approach works on diverse datasets in terms of content and scale, including a historical car dataset, a historical face dataset, a large worldwide street-view dataset, and an even larger scene dataset. Furthermore, our approach allows for translating visual elements across class labels and analyzing consistent changes.