mlsanityData to 7 Oct 2026

ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object

arXiv 2403.18775 ↗First posted 27 Mar 2024cs.CV cs.AI cs.LGcited by 22

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We establish rigorous benchmarks for visual perception robustness. Synthetic images such as ImageNet-C, ImageNet-9, and Stylized ImageNet provide specific type of evaluation over synthetic corruptions, backgrounds, and textures, yet those robustness benchmarks are restricted in specified variations and have low synthetic quality. In this work, we introduce generative model as a data source for synthesizing hard images that benchmark deep models' robustness. Leveraging diffusion models, we are able to generate images with more diversified backgrounds, textures, and materials than any prior work, where we term this benchmark as ImageNet-D. Experimental results show that ImageNet-D results in a significant accuracy drop to a range of vision models, from the standard ResNet visual classifier to the latest foundation models like CLIP and MiniGPT-4, significantly reducing their accuracy by up to 60%. Our work suggests that diffusion models can be an effective source to test vision models. The code and dataset are available at https://github.com/chenshuang-zhang/imagenetd.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 56 of its 65 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

56 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2016Densely Connected Convolutional Networkscited by 6,691
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  13. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  14. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  15. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  16. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  17. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  18. 2019Natural Adversarial Examplescited by 1,401
  19. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  20. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  21. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  22. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  23. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  24. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  25. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  26. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  27. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  28. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
  29. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  30. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  31. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  32. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  33. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  34. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  35. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  36. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  37. 2021Discrete Representations Strengthen Vision Transformer Robustnesscited by 34
  38. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 10.

22 papers.

Sort:

By area:

  1. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  2. 2025DASH: Detection and Assessment of Systematic Hallucinations of VLMscited by 14
  3. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  4. 2025Are We Done with Object-Centric Learning?cited by 10
  5. 2024Improving robustness to corruptions with multiplicative weight perturbationscited by 7
  6. 2024SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classificationcited by 5
  7. 2025GHOST: Hallucination-Inducing Image Generation for Multimodal LLMscited by 4
  8. 2025ForAug: Mitigating Biases in Image Classification via Controlled Image Compositionscited by 2
  9. 2025CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shiftscited by 2
  10. 2026Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editingcited by 2
  11. 2026RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenescited by 2
  12. 2026From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tamperingcited by 1
10 more
  1. 2025Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insightscited by 0
  2. 2025Do Vision Models Develop Human-Like Progressive Difficulty Understanding?cited by 0
  3. 2025On the Robustness of Human-Object Interaction Detection against Distribution Shiftcited by 0
  4. 2026PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Modelscited by 0
  5. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attackscited by 0
  6. 2026MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Spacecited by 0
  7. 2026Personalized Generative Models for Contextual Debiasingcited by 0
  8. 2026SABRE: Scalable and Automated Benchmarking of VLMs under Stresscited by 0
  9. 2026SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networkscited by 0
  10. 2026When Masking Helps or Hurts Robustness in Compressed CLIP: A Pre-Deployment Diagnosticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Benchmarking Robustness to Text-Guided Corruptions
  2. 2024Diffusion Models in Low-Level Vision: A Survey
  3. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images
  4. 2021Diffusion Models Beat GANs on Image Synthesis
  5. 2022Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance
  6. 2025Diffusion Models are Robust Pretrainers
  7. 2026LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models
  8. 2023A High-Quality Robust Diffusion Framework for Corrupted Dataset