# PathLDM: Text conditioned Latent Diffusion Model for Histopathology

arXiv: https://arxiv.org/abs/2309.00748 · HTML: https://mlsanity.com/paper/2309.00748/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 1 Sep 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [Classification Accuracy Score for Conditional Generative Models](https://mlsanity.com/paper/1905.10887/) (2019)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [Diffusion Probabilistic Models beat GANs on Medical Images](https://mlsanity.com/paper/2212.07501/) (2022)
- [A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Images](https://mlsanity.com/paper/2209.13167/) (2022)
- [ViT-DAE: Transformer-driven Diffusion Autoencoder for Histopathology Image Analysis](https://mlsanity.com/paper/2304.01053/) (2023)

## Built on by (most cited first, 30 of 52)

- [Learned representation-guided diffusion models for large-image generation](https://mlsanity.com/paper/2312.07330/) (2023)
- [FlowSDF: Flow Matching for Medical Image Segmentation Using Distance Transforms](https://mlsanity.com/paper/2405.18087/) (2024)
- [Adding Conditional Control to Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2406.12120/) (2024)
- [ZoomLDM: Latent Diffusion Model for multi-scale image generation](https://mlsanity.com/paper/2411.16969/) (2024)
- [TopoCellGen: Generating Histopathology Cell Topology with a Diffusion Model](https://mlsanity.com/paper/2412.06011/) (2024)
- [UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis](https://mlsanity.com/paper/2510.15710/) (2025)
- [PixCell: A generative foundation model for digital histopathology images](https://mlsanity.com/paper/2506.05127/) (2025)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions](https://mlsanity.com/paper/2506.23440/) (2025)
- [A Systematic Review of Generalization Research in Medical Image Classification](https://mlsanity.com/paper/2403.12167/) (2024)
- [Controllable and Efficient Multi-Class Pathology Nuclei Data Augmentation using Text-Conditioned Diffusion Models](https://mlsanity.com/paper/2407.14426/) (2024)
- [Co-synthesis of Histopathology Nuclei Image-Label Pairs using a Context-Conditioned Joint Diffusion Model](https://mlsanity.com/paper/2407.14434/) (2024)
- [Pixel super-resolved virtual staining of label-free tissue using diffusion models](https://mlsanity.com/paper/2410.20073/) (2024)
- [GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathology](https://mlsanity.com/paper/2504.01009/) (2025)
- [StainFuser: Controlling Diffusion for Faster Neural Style Transfer in Multi-Gigapixel Histology Images](https://mlsanity.com/paper/2403.09302/) (2024)
- [Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning](https://mlsanity.com/paper/2412.01672/) (2024)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion](https://mlsanity.com/paper/2505.20053/) (2025)
- [Histo-Diffusion: A Diffusion Super-Resolution Method for Digital Pathology with Comprehensive Quality Assessment](https://mlsanity.com/paper/2408.15218/) (2024)
- [LesionGen: A Concept-Guided Diffusion Model for Dermatology Image Synthesis](https://mlsanity.com/paper/2507.23001/) (2025)
- [Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology](https://mlsanity.com/paper/2509.17847/) (2025)
- [Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Control](https://mlsanity.com/paper/2512.21058/) (2025)
- [Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology](https://mlsanity.com/paper/2312.09792/) (2023)
- [PathSegDiff: Pathology Segmentation using Diffusion model representations](https://mlsanity.com/paper/2504.06950/) (2025)
- [SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models](https://mlsanity.com/paper/2603.17048/) (2026)
- [Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reports](https://mlsanity.com/paper/2405.14230/) (2024)
- [Mask-guided cross-image attention for zero-shot in-silico histopathologic image generation with a diffusion model](https://mlsanity.com/paper/2407.11664/) (2024)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)
- [Prototype-Guided Diffusion for Digital Pathology: Achieving Foundation Model Performance with Minimal Clinical Data](https://mlsanity.com/paper/2504.12351/) (2025)

## Abstract (arXiv)

To achieve high-quality results, diffusion models must be trained on large datasets. This can be notably prohibitive for models in specialized domains, such as computational pathology. Conditioning on labeled data is known to help in data-efficient model training. Therefore, histopathology reports, which are rich in valuable clinical information, are an ideal choice as guidance for a histopathology generative model. In this paper, we introduce PathLDM, the first text-conditioned Latent Diffusion Model tailored for generating high-quality histopathology images. Leveraging the rich contextual information provided by pathology text reports, our approach fuses image and textual data to enhance the generation process. By utilizing GPT's capabilities to distill and summarize complex text reports, we establish an effective conditioning mechanism. Through strategic conditioning and necessary architectural enhancements, we achieved a SoTA FID score of 7.64 for text-to-image generation on the TCGA-BRCA dataset, significantly outperforming the closest text-conditioned competitor with FID 30.1.