# DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2605.04593 · HTML: https://mlsanity.com/paper/2605.04593/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 54 of its 75 references that are in the map.
- First posted on arXiv 6 May 2026.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 54)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.10464/) (2018)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.02664/) (2022)
- [BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation](https://mlsanity.com/paper/2103.08907/) (2021)
- [Learning random-walk label propagation for weakly-supervised semantic segmentation](https://mlsanity.com/paper/1802.00470/) (2018)
- [Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast](https://mlsanity.com/paper/2110.07110/) (2021)
- [Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features](https://mlsanity.com/paper/1806.04659/) (2018)
- [Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter](https://mlsanity.com/paper/2309.02773/) (2023)
- [Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.00962/) (2022)
- [Cross Language Image Matching for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02668/) (2022)
- [Token Contrast for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2303.01267/) (2023)
- [Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.09653/) (2022)
- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)
- [Weakly Supervised Semantic Segmentation using Out-of-Distribution Data](https://mlsanity.com/paper/2203.03860/) (2022)

## Abstract (arXiv)

Weakly Supervised Semantic Segmentation (WSSS) with image-level labels typically leverages Class Activation Maps (CAMs) to achieve pixel-level predictions. Recently, Contrastive Language-Image Pre-training (CLIP) has been introduced to generate CAMs in WSSS. However, previous WSSS methods solely adopt CLIP's vision-language paired property for dense localization, neglecting its inherently limited dense knowledge across both visual and text modalities, which renders CAM generation suboptimal. In this work, we propose DiCLIP, a novel WSSS framework that leverages the generative diffusion model to enhance CLIP's dense knowledge across two modalities. Specifically, Visual Correlation Enhancement (VCE) and Text Semantic Augmentation (TSA) modules are proposed for dense prediction enhancement. To improve the spatial awareness of visual features, our VCE module utilizes diffusion's reliable spatial consistency to mitigate the over-smoothing issue in CLIP's attention. It designs the Attention Clustering Refinement (ACR) module to reliably extract diverse correlation maps from the diffusion model. The correlation maps act as a diversity bias for CLIP's self-attention, recursively pushing its visual features towards a more discriminative dense distribution. To augment the semantics of text embeddings, our TSA module argues that a single text modality is insufficient to encompass the variability of visual categories. Thus, we leverage diffusion's generative power to maintain a dynamic key-value cache model, shifting CAM generation from a patch-text matching mechanism to a novel visual knowledge retrieval paradigm. With these enhancements, DiCLIP not only outperforms state-of-the-art methods on PASCAL VOC and MS COCO but also significantly reduces training costs. Code is publicly available at https://github.com/zwyang6/DiCLIP.