# Dataset Enhancement with Instance-Level Augmentations

arXiv: https://arxiv.org/abs/2406.08249 · HTML: https://mlsanity.com/paper/2406.08249/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 48 of its 62 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)

## Built on by (most cited first, 23 of 23)

- [Personalized Representation from Personalized Generation](https://mlsanity.com/paper/2412.16156/) (2024)
- [ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detection](https://mlsanity.com/paper/2510.15783/) (2025)
- [S3OD: Towards Generalizable Salient Object Detection with Synthetic Data](https://mlsanity.com/paper/2510.21605/) (2025)
- [VGGHeads: 3D Multi Head Alignment with a Large-Scale Synthetic Dataset](https://mlsanity.com/paper/2407.18245/) (2024)
- [Imagining the Unseen: Generative Location Modeling for Object Placement](https://mlsanity.com/paper/2410.13564/) (2024)
- [AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors](https://mlsanity.com/paper/2601.20524/) (2026)
- [Generative augmentations for improved cardiac ultrasound segmentation using diffusion models](https://mlsanity.com/paper/2502.20100/) (2025)
- [Active Learning Inspired ControlNet Guidance for Augmenting Semantic Segmentation Datasets](https://mlsanity.com/paper/2503.09221/) (2025)
- [Gaussian Splatting is an Effective Data Generator for 3D Object Detection](https://mlsanity.com/paper/2504.16740/) (2025)
- [Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models](https://mlsanity.com/paper/2505.21574/) (2025)
- [7Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Models](https://mlsanity.com/paper/2508.12919/) (2025)
- [PoseSyn: Synthesizing Diverse 3D Pose Data from In-the-Wild 2D Data](https://mlsanity.com/paper/2503.13025/) (2025)
- [Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detection](https://mlsanity.com/paper/2504.17076/) (2025)
- [Understanding Trade offs When Conditioning Synthetic Data](https://mlsanity.com/paper/2507.02217/) (2025)
- [SemiISP/SemiIE: Semi-Supervised Image Signal Processor and Image Enhancement Leveraging One-to-Many Mapping sRGB-to-RAW](https://mlsanity.com/paper/2504.02345/) (2025)
- [On the Importance of Conditioning for Privacy-Preserving Data Augmentation](https://mlsanity.com/paper/2504.05849/) (2025)
- [ScoreMix: Synthetic Data Generation by Score Composition in Diffusion Models Improves Recognition](https://mlsanity.com/paper/2506.10226/) (2025)
- [A Framework for Low-Effort Training Data Generation for Urban Semantic Segmentation](https://mlsanity.com/paper/2510.11567/) (2025)
- [Test-Time Modification: Inverse Domain Transformation for Robust Perception](https://mlsanity.com/paper/2512.13454/) (2025)
- [R&D: Balancing Reliability and Diversity in Synthetic Data Augmentation for Semantic Segmentation](https://mlsanity.com/paper/2603.18427/) (2026)
- [VERIA: Verification-Centric Multimodal Instance Augmentation for Long-Tailed 3D Object Detection](https://mlsanity.com/paper/2603.24294/) (2026)
- [PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Models](https://mlsanity.com/paper/2603.28763/) (2026)
- [Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models](https://mlsanity.com/paper/2606.31603/) (2026)

## Abstract (arXiv)

We present a method for expanding a dataset by incorporating knowledge from the wide distribution of pre-trained latent diffusion models. Data augmentations typically incorporate inductive biases about the image formation process into the training (e.g. translation, scaling, colour changes, etc.). Here, we go beyond simple pixel transformations and introduce the concept of instance-level data augmentation by repainting parts of the image at the level of object instances. The method combines a conditional diffusion model with depth and edge maps control conditioning to seamlessly repaint individual objects inside the scene, being applicable to any segmentation or detection dataset. Used as a data augmentation method, it improves the performance and generalization of the state-of-the-art salient object detection, semantic segmentation and object detection models. By redrawing all privacy-sensitive instances (people, license plates, etc.), the method is also applicable for data anonymization. We also release fully synthetic and anonymized expansions for popular datasets: COCO, Pascal VOC and DUTS.