mlsanityData to 7 Oct 2026

Dataset Enhancement with Instance-Level Augmentations

arXiv 2406.08249 ↗First posted 12 Jun 2024cs.CV cs.LGcited by 23

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We present a method for expanding a dataset by incorporating knowledge from the wide distribution of pre-trained latent diffusion models. Data augmentations typically incorporate inductive biases about the image formation process into the training (e.g. translation, scaling, colour changes, etc.). Here, we go beyond simple pixel transformations and introduce the concept of instance-level data augmentation by repainting parts of the image at the level of object instances. The method combines a conditional diffusion model with depth and edge maps control conditioning to seamlessly repaint individual objects inside the scene, being applicable to any segmentation or detection dataset. Used as a data augmentation method, it improves the performance and generalization of the state-of-the-art salient object detection, semantic segmentation and object detection models. By redrawing all privacy-sensitive instances (people, license plates, etc.), the method is also applicable for data anonymization. We also release fully synthetic and anonymized expansions for popular datasets: COCO, Pascal VOC and DUTS.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 62 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

48 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models11 Aug
  2. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification17 Apr
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion7 Feb
  6. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clones16 Dec
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
28 more
  1. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  2. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  8. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  9. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  10. 2021Robust fine-tuning of zero-shot modelscited by 785
  11. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  12. 2015Holistically-Nested Edge Detectioncited by 590
  13. 2018Albumentations: fast and flexible image augmentationscited by 558
  14. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 526
  15. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
  16. 2020Virtual KITTI 2cited by 345
  17. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
  18. 2019Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relationscited by 245
  19. 2016Unrolled Generative Adversarial Networkscited by 208
  20. 2015Visual Saliency Based on Multiscale Deep Featurescited by 185
  21. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  22. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  23. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  24. 2019Towards High-Resolution Salient Object Detectioncited by 92
  25. 2021TRACER: Extreme Attention Guided Salient Object Tracing Networkcited by 32
  26. 2023LDFA: Latent Diffusion Face Anonymization for Self-driving Applicationscited by 21
  27. 2022Synthetic Data Supervised Salient Object Detectioncited by 15
  28. 2022HandsOff: Labeled Dataset Generation With No Additional Human Annotationscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024Personalized Representation from Personalized Generationcited by 9
  2. 2025ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detectioncited by 6
  3. 2025S3OD: Towards Generalizable Salient Object Detection with Synthetic Datacited by 5
  4. 2024VGGHeads: 3D Multi Head Alignment with a Large-Scale Synthetic Datasetcited by 4
  5. 2024Imagining the Unseen: Generative Location Modeling for Object Placementcited by 4
  6. 2025Generative augmentations for improved cardiac ultrasound segmentation using diffusion modelscited by 3
  7. 2025Active Learning Inspired ControlNet Guidance for Augmenting Semantic Segmentation Datasetscited by 2
  8. 2025Gaussian Splatting is an Effective Data Generator for 3D Object Detectioncited by 2
  9. 2025Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Modelscited by 2
  10. 20257Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Modelscited by 2
  11. 2026AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectorscited by 2
  12. 2025PoseSyn: Synthesizing Diverse 3D Pose Data from In-the-Wild 2D Datacited by 1
11 more
  1. 2025Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detectioncited by 1
  2. 2025Understanding Trade offs When Conditioning Synthetic Datacited by 1
  3. 2025SemiISP/SemiIE: Semi-Supervised Image Signal Processor and Image Enhancement Leveraging One-to-Many Mapping sRGB-to-RAWcited by 0
  4. 2025On the Importance of Conditioning for Privacy-Preserving Data Augmentationcited by 0
  5. 2025ScoreMix: Synthetic Data Generation by Score Composition in Diffusion Models Improves Recognitioncited by 0
  6. 2025A Framework for Low-Effort Training Data Generation for Urban Semantic Segmentationcited by 0
  7. 2025Test-Time Modification: Inverse Domain Transformation for Robust Perceptioncited by 0
  8. 2026R&D: Balancing Reliability and Diversity in Synthetic Data Augmentation for Semantic Segmentationcited by 0
  9. 2026VERIA: Verification-Centric Multimodal Instance Augmentation for Long-Tailed 3D Object Detectioncited by 0
  10. 2026PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Modelscited by 0
  11. 2026Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Simple Background Augmentation Method for Object Detection with Diffusion Model
  2. 2024Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection
  3. 2025Image compositing is all you need for data augmentation
  4. 2024Erase, then Redraw: A Novel Data Augmentation Approach for Free Space Detection Using Diffusion Model
  5. 2024Advances in Diffusion Models for Image Data Augmentation: A Review of Methods, Models, Evaluation Metrics and Future Research Directions
  6. 2025Prompt-Free Conditional Diffusion for Multi-object Image Augmentation
  7. 2024CIA: Controllable Image Augmentation Framework Based on Stable Diffusion
  8. 2022DiffusionInst: Diffusion Model for Instance Segmentation