# Style-Extracting Diffusion Models for Semi-Supervised Histopathology Segmentation

arXiv: https://arxiv.org/abs/2403.14429 · HTML: https://mlsanity.com/paper/2403.14429/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Swapping Autoencoder for Deep Image Manipulation](https://mlsanity.com/paper/2007.00653/) (2020)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Images](https://mlsanity.com/paper/2209.13167/) (2022)
- [Few-Shot Diffusion Models](https://mlsanity.com/paper/2205.15463/) (2022)
- [Pan-tumor CAnine cuTaneous Cancer Histology (CATCH) dataset](https://mlsanity.com/paper/2201.11446/) (2022)
- [Realistic Data Enrichment for Robust Image Segmentation in Histopathology](https://mlsanity.com/paper/2304.09534/) (2023)
- [Stain Style Transfer of Histopathology Images Via Structure-Preserved Generative Learning](https://mlsanity.com/paper/2007.12578/) (2020)
- [Improved HER2 Tumor Segmentation with Subtype Balancing using Deep Generative Networks](https://mlsanity.com/paper/2211.06150/) (2022)

## Built on by (most cited first, 3 of 3)

- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [PathoPainter: Augmenting Histopathology Segmentation via Tumor-aware Inpainting](https://mlsanity.com/paper/2503.04634/) (2025)
- [MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysis](https://mlsanity.com/paper/2602.17901/) (2026)

## Abstract (arXiv)

Deep learning-based image generation has seen significant advancements with diffusion models, notably improving the quality of generated images. Despite these developments, generating images with unseen characteristics beneficial for downstream tasks has received limited attention. To bridge this gap, we propose Style-Extracting Diffusion Models, featuring two conditioning mechanisms. Specifically, we utilize 1) a style conditioning mechanism which allows to inject style information of previously unseen images during image generation and 2) a content conditioning which can be targeted to a downstream task, e.g., layout for segmentation. We introduce a trainable style encoder to extract style information from images, and an aggregation block that merges style information from multiple style inputs. This architecture enables the generation of images with unseen styles in a zero-shot manner, by leveraging styles from unseen images, resulting in more diverse generations. In this work, we use the image layout as target condition and first show the capability of our method on a natural image dataset as a proof-of-concept. We further demonstrate its versatility in histopathology, where we combine prior knowledge about tissue composition and unannotated data to create diverse synthetic images with known layouts. This allows us to generate additional synthetic data to train a segmentation network in a semi-supervised fashion. We verify the added value of the generated images by showing improved segmentation results and lower performance variability between patients when synthetic images are included during segmentation training. Our code will be made publicly available at [LINK].