# Image Augmentation with Controlled Diffusion for Weakly-Supervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2310.09760 · HTML: https://mlsanity.com/paper/2310.09760/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 15 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation](https://mlsanity.com/paper/1603.06098/) (2016)
- [Weakly-Supervised Semantic Segmentation via Sub-category Exploration](https://mlsanity.com/paper/2008.01183/) (2020)
- [Multi-class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02891/) (2022)
- [Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.02664/) (2022)
- [Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02909/) (2022)
- [Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast](https://mlsanity.com/paper/2110.07110/) (2021)
- [Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation](https://mlsanity.com/paper/2210.17400/) (2022)

## Built on by (most cited first, 19 of 19)

- [POSTA: A Go-to Framework for Customized Artistic Poster Generation](https://mlsanity.com/paper/2503.14908/) (2025)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Prompt Categories Cluster for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2412.13823/) (2024)
- [Adaptive Patch Contrast for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2407.10649/) (2024)
- [Image Augmentation Agent for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2412.20439/) (2024)
- [SMAFormer: Synergistic Multi-Attention Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2409.00346/) (2024)
- [Test-Time Intensity Consistency Adaptation for Shadow Detection](https://mlsanity.com/paper/2410.07695/) (2024)
- [Uncertainty-Participation Context Consistency Learning for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2412.17331/) (2024)
- [Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning](https://mlsanity.com/paper/2511.11607/) (2025)
- [CIA: Controllable Image Augmentation Framework Based on Stable Diffusion](https://mlsanity.com/paper/2411.16128/) (2024)
- [Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion](https://mlsanity.com/paper/2504.03135/) (2025)
- [MSA-UNet3+: Multi-Scale Attention UNet3+ with New Supervised Prototypical Contrastive Loss for Coronary DSA Image Segmentation](https://mlsanity.com/paper/2504.05184/) (2025)
- [Contrastive Prompt Clustering for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2508.17009/) (2025)
- [SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filtering](https://mlsanity.com/paper/2509.18603/) (2025)
- [SynthSeg-Agents: Multi-Agent Synthetic Data Generation for Zero-Shot Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2512.15310/) (2025)
- [Medical Visual Prompting (MVP): A Unified Framework for Versatile and High-Quality Medical Image Segmentation](https://mlsanity.com/paper/2404.01127/) (2024)
- [Multi-Object Grounding via Hierarchical Contrastive Siamese Transformers](https://mlsanity.com/paper/2504.10048/) (2025)
- [Context Patch Fusion With Class Token Enhancement for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2601.14718/) (2026)
- [DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2605.04593/) (2026)

## Abstract (arXiv)

Weakly-supervised semantic segmentation (WSSS), which aims to train segmentation models solely using image-level labels, has achieved significant attention. Existing methods primarily focus on generating high-quality pseudo labels using available images and their image-level labels. However, the quality of pseudo labels degrades significantly when the size of available dataset is limited. Thus, in this paper, we tackle this problem from a different view by introducing a novel approach called Image Augmentation with Controlled Diffusion (IACD). This framework effectively augments existing labeled datasets by generating diverse images through controlled diffusion, where the available images and image-level labels are served as the controlling information. Moreover, we also propose a high-quality image selection strategy to mitigate the potential noise introduced by the randomness of diffusion models. In the experiments, our proposed IACD approach clearly surpasses existing state-of-the-art methods. This effect is more obvious when the amount of available data is small, demonstrating the effectiveness of our method.