# MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer

arXiv: https://arxiv.org/abs/2301.11798 · HTML: https://mlsanity.com/paper/2301.11798/

## Key facts

- Cited by 124 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 19 Jan 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 29 of 29)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [Diffusion Models for Implicit Image Segmentation Ensembles](https://mlsanity.com/paper/2112.03145/) (2021)
- [MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model](https://mlsanity.com/paper/2211.00611/) (2022)
- [Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction](https://mlsanity.com/paper/2001.00208/) (2020)
- [Diffusion Adversarial Representation Learning for Self-supervised Vessel Segmentation](https://mlsanity.com/paper/2209.14566/) (2022)
- [Boundary and Entropy-driven Adversarial Learning for Fundus Image Segmentation](https://mlsanity.com/paper/1906.11143/) (2019)
- [Accelerating Diffusion Models via Pre-segmentation Diffusion Sampling for Medical Image Segmentation](https://mlsanity.com/paper/2210.17408/) (2022)
- [Automated Skin Lesion Classification Using Ensemble of Deep Neural Networks in ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection Challenge](https://mlsanity.com/paper/1901.10802/) (2019)
- [Leveraging Undiagnosed Data for Glaucoma Classification with Teacher-Student Learning](https://mlsanity.com/paper/2007.11355/) (2020)
- [Learning self-calibrated optic disc and cup segmentation from multi-rater annotations](https://mlsanity.com/paper/2206.05092/) (2022)
- [SeATrans: Learning Segmentation-Assisted diagnosis model via Transformer](https://mlsanity.com/paper/2206.05763/) (2022)
- [Universal, transferable and targeted adversarial attacks](https://mlsanity.com/paper/1908.11332/) (2019)
- [Improved HER2 Tumor Segmentation with Subtype Balancing using Deep Generative Networks](https://mlsanity.com/paper/2211.06150/) (2022)

## Built on by (most cited first, 30 of 124)

- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Stochastic Segmentation with Conditional Categorical Diffusion Models](https://mlsanity.com/paper/2303.08888/) (2023)
- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process](https://mlsanity.com/paper/2312.12425/) (2023)
- [Diffusion Model for Camouflaged Object Detection](https://mlsanity.com/paper/2308.00303/) (2023)
- [How to Protect Copyright Data in Optimization of Large Language Models?](https://mlsanity.com/paper/2308.12247/) (2023)
- [FlowSDF: Flow Matching for Medical Image Segmentation Using Distance Transforms](https://mlsanity.com/paper/2405.18087/) (2024)
- [DiffBoost: Enhancing Medical Image Segmentation via Text-Guided Diffusion Model](https://mlsanity.com/paper/2310.12868/) (2023)
- [HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization](https://mlsanity.com/paper/2405.19751/) (2024)
- [Stable Diffusion Segmentation for Biomedical Images with Single-step Reverse Process](https://mlsanity.com/paper/2406.18361/) (2024)
- [Denoising Diffusion Semantic Segmentation with Mask Prior Modeling](https://mlsanity.com/paper/2306.01721/) (2023)
- [LLCaps: Learning to Illuminate Low-Light Capsule Endoscopy with Curved Wavelet Attention and Reverse Diffusion](https://mlsanity.com/paper/2307.02452/) (2023)
- [FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis](https://mlsanity.com/paper/2406.01264/) (2024)
- [HiDiff: Hybrid Diffusion Framework for Medical Image Segmentation](https://mlsanity.com/paper/2407.03548/) (2024)
- [DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing](https://mlsanity.com/paper/2409.07756/) (2024)
- [DermoSegDiff: A Boundary-aware Segmentation Diffusion Model for Skin Lesion Delineation](https://mlsanity.com/paper/2308.02959/) (2023)
- [A Survey of Emerging Applications of Diffusion Probabilistic Models in MRI](https://mlsanity.com/paper/2311.11383/) (2023)
- [Conditional Diffusion Models are Medical Image Classifiers that Provide Explainability and Uncertainty for Free](https://mlsanity.com/paper/2502.03687/) (2025)
- [Recent Advances in Medical Imaging Segmentation: A Survey](https://mlsanity.com/paper/2505.09274/) (2025)
- [MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing](https://mlsanity.com/paper/2506.01921/) (2025)
- [Symmetrical Flow Matching: Unified Image Generation, Segmentation, and Classification with Score-Based Generative Models](https://mlsanity.com/paper/2506.10634/) (2025)
- [Generative Medical Segmentation](https://mlsanity.com/paper/2403.18198/) (2024)
- [Diffuse-UDA: Addressing Unsupervised Domain Adaptation in Medical Image Segmentation with Appearance and Structure Aligned Diffusion Models](https://mlsanity.com/paper/2408.05985/) (2024)
- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [Applications of Large Models in Medicine](https://mlsanity.com/paper/2502.17132/) (2025)
- [A Recycling Training Strategy for Medical Image Segmentation with Diffusion Denoising Models](https://mlsanity.com/paper/2308.16355/) (2023)
- [SegICL: A Multimodal In-context Learning Framework for Enhanced Segmentation in Medical Imaging](https://mlsanity.com/paper/2403.16578/) (2024)
- [Lane Graph Extraction from Aerial Imagery via Lane Segmentation Refinement with Diffusion Models](https://mlsanity.com/paper/2405.00620/) (2024)

## Abstract (arXiv)

The Diffusion Probabilistic Model (DPM) has recently gained popularity in the field of computer vision, thanks to its image generation applications, such as Imagen, Latent Diffusion Models, and Stable Diffusion, which have demonstrated impressive capabilities and sparked much discussion within the community. Recent investigations have further unveiled the utility of DPM in the domain of medical image analysis, as underscored by the commendable performance exhibited by the medical image segmentation model across various tasks. Although these models were originally underpinned by a UNet architecture, there exists a potential avenue for enhancing their performance through the integration of vision transformer mechanisms. However, we discovered that simply combining these two models resulted in subpar performance. To effectively integrate these two cutting-edge techniques for the Medical image segmentation, we propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities. Through comprehensive evaluation, our approach demonstrates superiority over prior state-of-the-art (SOTA) methodologies. Code is released at https://github.com/KidsWithTokens/MedSegDiff