# Image Augmentation Agent for Weakly Supervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2412.20439 · HTML: https://mlsanity.com/paper/2412.20439/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 29 Dec 2024.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation](https://mlsanity.com/paper/1603.06098/) (2016)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [Weakly-Supervised Semantic Segmentation via Sub-category Exploration](https://mlsanity.com/paper/2008.01183/) (2020)
- [Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2007.01947/) (2020)
- [Multi-class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02891/) (2022)
- [Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2105.08965/) (2021)
- [Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.02664/) (2022)
- [Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02909/) (2022)
- [Token Contrast for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2303.01267/) (2023)
- [Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation](https://mlsanity.com/paper/2210.17400/) (2022)
- [Controllable Generation from Pre-trained Language Models via Inverse Prompting](https://mlsanity.com/paper/2103.10685/) (2021)
- [Self Correspondence Distillation for End-to-End Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2302.13765/) (2023)
- [Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching](https://mlsanity.com/paper/2112.09459/) (2021)
- [Image Augmentation with Controlled Diffusion for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2310.09760/) (2023)
- [SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2401.11719/) (2024)
- [Dual-Hybrid Attention Network for Specular Highlight Removal](https://mlsanity.com/paper/2407.12255/) (2024)
- [Adaptive Patch Contrast for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2407.10649/) (2024)

## Built on by (most cited first, 15 of 15)

- [Hybrid Generative Fusion for Efficient and Privacy-Preserving Face Recognition Dataset Generation](https://mlsanity.com/paper/2508.10672/) (2025)
- [Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning](https://mlsanity.com/paper/2511.11607/) (2025)
- [Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion](https://mlsanity.com/paper/2504.03135/) (2025)
- [MSA-UNet3+: Multi-Scale Attention UNet3+ with New Supervised Prototypical Contrastive Loss for Coronary DSA Image Segmentation](https://mlsanity.com/paper/2504.05184/) (2025)
- [Contrastive Prompt Clustering for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2508.17009/) (2025)
- [SynthSeg-Agents: Multi-Agent Synthetic Data Generation for Zero-Shot Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2512.15310/) (2025)
- [Multi-Object Grounding via Hierarchical Contrastive Siamese Transformers](https://mlsanity.com/paper/2504.10048/) (2025)
- [DFEN: Dual Feature Equalization Network for Medical Image Segmentation](https://mlsanity.com/paper/2505.05913/) (2025)
- [SFormer: SNR-guided Transformer for Underwater Image Enhancement from the Frequency Domain](https://mlsanity.com/paper/2508.18664/) (2025)
- [A Comprehensive Analysis of Churn Prediction in Telecommunications Using Machine Learning](https://mlsanity.com/paper/2509.22654/) (2025)
- [Forestpest-YOLO: A High-Performance Detection Framework for Small Forestry Pests](https://mlsanity.com/paper/2510.00547/) (2025)
- [Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarization](https://mlsanity.com/paper/2510.17501/) (2025)
- [Context Patch Fusion With Class Token Enhancement for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2601.14718/) (2026)
- [Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection](https://mlsanity.com/paper/2603.07486/) (2026)
- [SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation](https://mlsanity.com/paper/2607.08161/) (2026)

## Abstract (arXiv)

Weakly-supervised semantic segmentation (WSSS) has achieved remarkable progress using only image-level labels. However, most existing WSSS methods focus on designing new network structures and loss functions to generate more accurate dense labels, overlooking the limitations imposed by fixed datasets, which can constrain performance improvements. We argue that more diverse trainable images provides WSSS richer information and help model understand more comprehensive semantic pattern. Therefore in this paper, we introduce a novel approach called Image Augmentation Agent (IAA) which shows that it is possible to enhance WSSS from data generation perspective. IAA mainly design an augmentation agent that leverages large language models (LLMs) and diffusion models to automatically generate additional images for WSSS. In practice, to address the instability in prompt generation by LLMs, we develop a prompt self-refinement mechanism. It allow LLMs to re-evaluate the rationality of generated prompts to produce more coherent prompts. Additionally, we insert an online filter into diffusion generation process to dynamically ensure the quality and balance of generated images. Experimental results show that our method significantly surpasses state-of-the-art WSSS approaches on the PASCAL VOC 2012 and MS COCO 2014 datasets.