mlsanityData to 14 Oct 2026

Invariant Learning via Diffusion Dreamed Distribution Shifts

arXiv 2211.10370 ↗First posted 18 Nov 2022cs.CV cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Though the background is an important signal for image classification, over reliance on it can lead to incorrect predictions when spurious correlations between foreground and background are broken at test time. Training on a dataset where these correlations are unbiased would lead to more robust models. In this paper, we propose such a dataset called Diffusion Dreamed Distribution Shifts (D3S). D3S consists of synthetic images generated through StableDiffusion using text prompts and image guides obtained by pasting a sample foreground image onto a background template image. Using this scalable approach we generate 120K images of objects from all 1000 ImageNet classes in 10 diverse backgrounds. Due to the incredible photorealism of the diffusion model, our images are much closer to natural images than previous synthetic datasets. D3S contains a validation set of more than 17K images whose labels are human-verified in an MTurk study. Using the validation set, we evaluate several popular DNN image classifiers and find that the classification performance of models generally suffers on our background diverse images. Next, we leverage the foreground & background labels in D3S to learn a foreground (background) representation that is invariant to changes in background (foreground) by penalizing the mutual information between the foreground (background) features and the background (foreground) labels. Linear classifiers trained on these features to predict foreground (background) from foreground (background) have high accuracies at 82.9% (93.8%), while classifiers that predict these labels from background and foreground have a much lower accuracy of 2.4% and 45.6% respectively. This suggests that our foreground and background features are well disentangled. We further test the efficacy of these representations by training classifiers on a task with strong spurious correlations.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 58 of its 73 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

58 of 73 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2016Densely Connected Convolutional Networkscited by 6,720
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  13. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  14. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  15. 2017Improved Training of Wasserstein GANscited by 2,235
  16. 2019Searching for MobileNetV3cited by 1,994
  17. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  18. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  19. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  20. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  21. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  22. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  23. 2018Recognition in Terra Incognitacited by 658
  24. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  25. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  26. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  27. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  28. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  29. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  30. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  31. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  32. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  33. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  34. 2018Adversarial Attacks and Defences Competitioncited by 96
  35. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 90
  36. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  37. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  38. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 9; 2024: 1.

10 papers.

  1. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  2. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 43
  3. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  4. 2023RadEdit: stress-testing biomedical vision models via diffusion image editingcited by 28
  5. 2023Identification of Systematic Errors of Image Classifiers on Rare Subgroupscited by 17
  6. 2023PRIME: Prioritizing Interpretability in Failure Mode Extractioncited by 9
  7. 2024Towards Formalizing Spuriousness of Biased Datasets Using Partial Information Decompositioncited by 6
  8. 2023ScribbleGen: Generative Data Augmentation Improves Scribble-supervised Semantic Segmentationcited by 5
  9. 2023Identifying and Mitigating Model Failures through Few-shot CLIP-aided Diffusion Generationcited by 3
  10. 2023Adapting Self-Supervised Representations to Multi-Domain Setupscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Models in Low-Level Vision: A Survey
  2. 2023Diversity and Diffusion: Observations on Synthetic Image Distributions with Stable Diffusion
  3. 2024Diffusion Models Need Visual Priors for Image Generation
  4. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object
  5. 2023Diffusion Noise Feature: Accurate and Fast Generated Image Detection
  6. 2026Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning
  7. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification
  8. 2024Synthesis and Perceptual Scaling of High Resolution Naturalistic Images Using Stable Diffusion