mlsanityData to 14 Oct 2026

Dataset Augmentation by Mixing Visual Concepts

arXiv 2412.15358 ↗First posted 19 Dec 2024cs.CVcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

This paper proposes a dataset augmentation method by fine-tuning pre-trained diffusion models. Generating images using a pre-trained diffusion model with textual conditioning often results in domain discrepancy between real data and generated images. We propose a fine-tuning approach where we adapt the diffusion model by conditioning it with real images and novel text embeddings. We introduce a unique procedure called Mixing Visual Concepts (MVC) where we create novel text embeddings from image captions. The MVC enables us to generate multiple images which are diverse and yet similar to the real data enabling us to perform effective dataset augmentation. We perform comprehensive qualitative and quantitative evaluations with the proposed dataset augmentation approach showcasing both coarse-grained and finegrained changes in generated images. Our approach outperforms state-of-the-art augmentation techniques on benchmark classification tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 37 of its 49 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

37 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification17 Apr
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2023Effective Data Augmentation With Diffusion Models7 Feb
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2016Wide Residual Networkscited by 3,070
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  11. 2015Variational Inference with Normalizing Flowscited by 2,170
  12. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
19 more
  1. 2022Diffusion Models in Vision: A Surveycited by 1,015
  2. 2019An Introduction to Variational Autoencoderscited by 1,000
  3. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  4. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  5. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  6. 2020Jukebox: A Generative Model for Musiccited by 542
  7. 2017VisDA: The Visual Domain Adaptation Challengecited by 435
  8. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  9. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  10. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 252
  11. 2019Fast AutoAugmentcited by 243
  12. 2020A Review on Generative Adversarial Networks: Algorithms, Theory, and Applicationscited by 209
  13. 2017Dataset Augmentation in Feature Spacecited by 124
  14. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  15. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  16. 2019This dataset does not exist: training models from generated imagescited by 65
  17. 2017Smart Augmentation - Learning an Optimal Data Augmentation Strategycited by 54
  18. 2017A Bayesian Data Augmentation Approach for Learning Deep Modelscited by 50
  19. 2016Deep Directed Generative Models with Energy-Based Probability Estimationcited by 34

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Semantic Augmentation in Images using Language
  2. 2025Context-guided Responsible Data Augmentation with Diffusion Models
  3. 2022Cap2Aug: Caption guided Image to Image data Augmentation
  4. 2025SGD-Mix: Enhancing Domain-Specific Image Classification with Label-Preserving Data Augmentation
  5. 2024Advances in Diffusion Models for Image Data Augmentation: A Review of Methods, Models, Evaluation Metrics and Future Research Directions
  6. 2024Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model
  7. 2023Detail Reinforcement Diffusion Model: Augmentation Fine-Grained Visual Categorization in Few-Shot Conditions
  8. 2023Multimodal Data Augmentation for Image Captioning using Diffusion Models