# Unsupervised Discovery of 3D Hierarchical Structure with Generative Diffusion Features

arXiv: https://arxiv.org/abs/2305.00067 · HTML: https://mlsanity.com/paper/2305.00067/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 26 of 26)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [Diffusion Models for Medical Anomaly Detection](https://mlsanity.com/paper/2203.04306/) (2022)
- [Diffusion Models for Implicit Image Segmentation Ensembles](https://mlsanity.com/paper/2112.03145/) (2021)
- [MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model](https://mlsanity.com/paper/2211.00611/) (2022)
- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [Data augmentation using learned transformations for one-shot medical image segmentation](https://mlsanity.com/paper/1902.09383/) (2019)
- [Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach](https://mlsanity.com/paper/1703.10908/) (2017)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models](https://mlsanity.com/paper/2206.03461/) (2022)
- [Deep Feature Factorization For Concept Discovery](https://mlsanity.com/paper/1806.10206/) (2018)
- [Unsupervised Part Discovery from Contrastive Reconstruction](https://mlsanity.com/paper/2111.06349/) (2021)
- [Diffusion Adversarial Representation Learning for Self-supervised Vessel Segmentation](https://mlsanity.com/paper/2209.14566/) (2022)
- [What is Healthy? Generative Counterfactual Diffusion for Lesion Localization](https://mlsanity.com/paper/2207.12268/) (2022)
- [Capturing implicit hierarchical structure in 3D biomedical images with self-supervised hyperbolic representations](https://mlsanity.com/paper/2012.01644/) (2020)
- [Decoder Denoising Pretraining for Semantic Segmentation](https://mlsanity.com/paper/2205.11423/) (2022)
- [Unsupervised Denoising of Retinal OCT with Diffusion Probabilistic Model](https://mlsanity.com/paper/2201.11760/) (2022)
- [Unsupervised Segmentation of Hyperspectral Images Using 3D Convolutional Autoencoders](https://mlsanity.com/paper/1907.08870/) (2019)
- [Unsupervised Segmentation of 3D Medical Images Based on Clustering and Deep Representation Learning](https://mlsanity.com/paper/1804.03830/) (2018)

## Built on by (most cited first, 4 of 4)

- [A Survey of Emerging Applications of Diffusion Probabilistic Models in MRI](https://mlsanity.com/paper/2311.11383/) (2023)
- [Conditional Generation from Unconditional Diffusion Models using Denoiser Representations](https://mlsanity.com/paper/2306.01900/) (2023)
- [Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Models](https://mlsanity.com/paper/2506.02419/) (2025)
- [TextDiffSeg: Text-guided Latent Diffusion Model for 3d Medical Images Segmentation](https://mlsanity.com/paper/2504.11825/) (2025)

## Abstract (arXiv)

Inspired by recent findings that generative diffusion models learn semantically meaningful representations, we use them to discover the intrinsic hierarchical structure in biomedical 3D images using unsupervised segmentation. We show that features of diffusion models from different stages of a U-Net-based ladder-like architecture capture different hierarchy levels in 3D biomedical images. We design three losses to train a predictive unsupervised segmentation network that encourages the decomposition of 3D volumes into meaningful nested subvolumes that represent a hierarchy. First, we pretrain 3D diffusion models and use the consistency of their features across subvolumes. Second, we use the visual consistency between subvolumes. Third, we use the invariance to photometric augmentations as a regularizer. Our models achieve better performance than prior unsupervised structure discovery approaches on challenging biologically-inspired synthetic datasets and on a real-world brain tumor MRI dataset.