# Controllable and Efficient Multi-Class Pathology Nuclei Data Augmentation using Text-Conditioned Diffusion Models

arXiv: https://arxiv.org/abs/2407.14426 · HTML: https://mlsanity.com/paper/2407.14426/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 25 of 25)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [Lizard: A Large-Scale Dataset for Colonic Nuclear Instance Segmentation and Classification](https://mlsanity.com/paper/2108.11195/) (2021)
- [PathLDM: Text conditioned Latent Diffusion Model for Histopathology](https://mlsanity.com/paper/2309.00748/) (2023)
- [Diffusion-based Data Augmentation for Nuclei Image Segmentation](https://mlsanity.com/paper/2310.14197/) (2023)
- [DiffMix: Diffusion Model-based Data Synthesis for Nuclei Segmentation and Classification in Imbalanced Pathology Image Datasets](https://mlsanity.com/paper/2306.14132/) (2023)
- [NASDM: Nuclei-Aware Semantic Histopathology Image Generation Using Diffusion Models](https://mlsanity.com/paper/2303.11477/) (2023)
- [Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesis](https://mlsanity.com/paper/2308.08157/) (2023)
- [GradMix for nuclei segmentation and classification in imbalanced pathology image datasets](https://mlsanity.com/paper/2210.12938/) (2022)
- [SIAN: Style-Guided Instance-Adaptive Normalization for Multi-Organ Histopathology Image Synthesis](https://mlsanity.com/paper/2209.02412/) (2022)

## Built on by (most cited first, 8 of 8)

- [PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions](https://mlsanity.com/paper/2506.23440/) (2025)
- [Pixel super-resolved virtual staining of label-free tissue using diffusion models](https://mlsanity.com/paper/2410.20073/) (2024)
- [LCMem: A Universal Model for Robust Image Memorization Detection](https://mlsanity.com/paper/2512.14421/) (2025)
- [Synthetic Data Augmentation using Pre-trained Diffusion Models for Long-tailed Food Image Classification](https://mlsanity.com/paper/2506.01368/) (2025)
- [Enabling PSO-Secure Synthetic Data Sharing Using Diversity-Aware Diffusion Models](https://mlsanity.com/paper/2506.17975/) (2025)
- [CLIP-IT: CLIP-based Pairing for Histology Images Classification](https://mlsanity.com/paper/2504.16181/) (2025)
- [Instance Migration Diffusion for Nuclear Instance Segmentation in Pathology](https://mlsanity.com/paper/2504.01577/) (2025)
- [MSDM: Generating Task-Specific Pathology Images with a Multimodal Conditioned Diffusion Model for Cell and Nuclei Segmentation](https://mlsanity.com/paper/2510.09121/) (2025)

## Abstract (arXiv)

In the field of computational pathology, deep learning algorithms have made significant progress in tasks such as nuclei segmentation and classification. However, the potential of these advanced methods is limited by the lack of available labeled data. Although image synthesis via recent generative models has been actively explored to address this challenge, existing works have barely addressed label augmentation and are mostly limited to single-class and unconditional label generation. In this paper, we introduce a novel two-stage framework for multi-class nuclei data augmentation using text-conditional diffusion models. In the first stage, we innovate nuclei label synthesis by generating multi-class semantic labels and corresponding instance maps through a joint diffusion model conditioned by text prompts that specify the label structure information. In the second stage, we utilize a semantic and text-conditional latent diffusion model to efficiently generate high-quality pathology images that align with the generated nuclei label images. We demonstrate the effectiveness of our method on large and diverse pathology nuclei datasets, with evaluations including qualitative and quantitative analyses, as well as assessments of downstream tasks.