# Context-guided Responsible Data Augmentation with Diffusion Models

arXiv: https://arxiv.org/abs/2503.10687 · HTML: https://mlsanity.com/paper/2503.10687/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 46 of its 58 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)
- [StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners](https://mlsanity.com/paper/2306.00984/) (2023)
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](https://mlsanity.com/paper/2305.07027/) (2023)
- [FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization](https://mlsanity.com/paper/2303.14189/) (2023)
- [SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization](https://mlsanity.com/paper/2006.01791/) (2020)
- [Rich Human Feedback for Text-to-Image Generation](https://mlsanity.com/paper/2312.10240/) (2023)
- [PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures](https://mlsanity.com/paper/2112.05135/) (2021)
- [Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity](https://mlsanity.com/paper/2102.03065/) (2021)
- [DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing](https://mlsanity.com/paper/2402.02583/) (2024)
- [TransMix: Attend to Mix for Vision Transformers](https://mlsanity.com/paper/2111.09833/) (2021)
- [DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models](https://mlsanity.com/paper/2405.14881/) (2024)
- [ResizeMix: Mixing Data with Preserved Object Information and True Labels](https://mlsanity.com/paper/2012.11101/) (2020)
- [AutoMix: Unveiling the Power of Mixup for Stronger Classifiers](https://mlsanity.com/paper/2103.13027/) (2021)
- [DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations](https://mlsanity.com/paper/2403.06951/) (2024)

## Built on by (most cited first, 9 of 9)

- [Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models](https://mlsanity.com/paper/2505.21574/) (2025)
- [S2-FracMix: Label-Preserving Self-Saliency Mixup Augmentation](https://mlsanity.com/paper/2606.25784/) (2026)
- [InstructMixup: Instruction-Guided Salient Patch Editing for Robust Data Augmentation](https://mlsanity.com/paper/2607.19324/) (2026)
- [Generating Synthetic Data via Augmentations for Improved Facial Resemblance in DreamBooth and InstantID](https://mlsanity.com/paper/2505.03557/) (2025)
- [SimDiffRec: Semantic Similarity-Guided Diffusion for Contrastive Sequential Recommendation](https://mlsanity.com/paper/2507.11866/) (2025)
- [Beyond Cropping and Rotation: Automated Evolution of Powerful Task-Specific Augmentations with Generative Models](https://mlsanity.com/paper/2602.03123/) (2026)
- [EdgeDAM: Real-time Object Tracking for Mobile Devices](https://mlsanity.com/paper/2603.05463/) (2026)
- [Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment](https://mlsanity.com/paper/2606.30262/) (2026)
- [AGVBench: A Reliability-Oriented Benchmark of Data Augmentation for Vein Recognition](https://mlsanity.com/paper/2607.02271/) (2026)

## Abstract (arXiv)

Generative diffusion models offer a natural choice for data augmentation when training complex vision models. However, ensuring reliability of their generative content as augmentation samples remains an open challenge. Despite a number of techniques utilizing generative images to strengthen model training, it remains unclear how to utilize the combination of natural and generative images as a rich supervisory signal for effective model induction. In this regard, we propose a text-to-image (T2I) data augmentation method, named DiffCoRe-Mix, that computes a set of generative counterparts for a training sample with an explicitly constrained diffusion model that leverages sample-based context and negative prompting for a reliable augmentation sample generation. To preserve key semantic axes, we also filter out undesired generative samples in our augmentation process. To that end, we propose a hard-cosine filtration in the embedding space of CLIP. Our approach systematically mixes the natural and generative images at pixel and patch levels. We extensively evaluate our technique on ImageNet-1K,Tiny ImageNet-200, CIFAR-100, Flowers102, CUB-Birds, Stanford Cars, and Caltech datasets, demonstrating a notable increase in performance across the board, achieving up to ~ 3% absolute gain for top-1 accuracy over the state-of-the-art methods, while showing comparable computational overhead. Our code is publicly available at https://github.com/khawar-islam/DiffCoRe-Mix