# ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing

arXiv: https://arxiv.org/abs/2303.17096 · HTML: https://mlsanity.com/paper/2303.17096/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)

## Built on by (most cited first, 30 of 30)

- [LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images](https://mlsanity.com/paper/2305.19164/) (2023)
- [A Sober Look at the Robustness of CLIPs to Spurious Features](https://mlsanity.com/paper/2403.11497/) (2024)
- [UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling](https://mlsanity.com/paper/2408.04810/) (2024)
- [RadEdit: stress-testing biomedical vision models via diffusion image editing](https://mlsanity.com/paper/2312.12865/) (2023)
- [Reliability in Semantic Segmentation: Can We Use Synthetic Data?](https://mlsanity.com/paper/2312.09231/) (2023)
- [ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object](https://mlsanity.com/paper/2403.18775/) (2024)
- [Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains](https://mlsanity.com/paper/2404.15882/) (2024)
- [A Survey on the Robustness of Computer Vision Models against Common Corruptions](https://mlsanity.com/paper/2305.06024/) (2023)
- [Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors](https://mlsanity.com/paper/2411.07472/) (2024)
- [IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers](https://mlsanity.com/paper/2310.04780/) (2023)
- [Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?](https://mlsanity.com/paper/2307.13136/) (2023)
- [RoDLA: Benchmarking the Robustness of Document Layout Analysis Models](https://mlsanity.com/paper/2403.14442/) (2024)
- [Benchmarking Segmentation Models with Mask-Preserved Attribute Editing](https://mlsanity.com/paper/2403.01231/) (2024)
- [ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes](https://mlsanity.com/paper/2403.04701/) (2024)
- [Open-Vocabulary Object Detectors: Robustness Challenges under Distribution Shifts](https://mlsanity.com/paper/2405.14874/) (2024)
- [RoHOI: Robustness Benchmark for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.09111/) (2025)
- [Processing and acquisition traces in visual encoders: What does CLIP know about your camera?](https://mlsanity.com/paper/2508.10637/) (2025)
- [M"obius Transform for Mitigating Perspective Distortions in Representation Learning](https://mlsanity.com/paper/2405.02296/) (2024)
- [ForAug: Mitigating Biases in Image Classification via Controlled Image Compositions](https://mlsanity.com/paper/2503.09399/) (2025)
- [CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shifts](https://mlsanity.com/paper/2507.17651/) (2025)
- [Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editing](https://mlsanity.com/paper/2603.01535/) (2026)
- [Reinforcing Pre-trained Models Using Counterfactual Images](https://mlsanity.com/paper/2406.13316/) (2024)
- [Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimators](https://mlsanity.com/paper/2408.16536/) (2024)
- [CHAOS: Chart Analysis with Outlier Samples](https://mlsanity.com/paper/2505.17235/) (2025)
- [Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs](https://mlsanity.com/paper/2507.00754/) (2025)
- [Identification of Fine-grained Systematic Errors via Controlled Scene Generation](https://mlsanity.com/paper/2404.07045/) (2024)
- [LCM: Log Conformal Maps for Robust Representation Learning to Mitigate Perspective Distortion](https://mlsanity.com/paper/2410.03686/) (2024)
- [Polyp-E: Benchmarking the Robustness of Deep Segmentation Models via Polyp Editing](https://mlsanity.com/paper/2410.16732/) (2024)
- [MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space](https://mlsanity.com/paper/2605.20549/) (2026)
- [The JEPA Predictor: A Transferable Operator for Occluded Feature Completion](https://mlsanity.com/paper/2607.16274/) (2026)

## Abstract (arXiv)

Recent studies have shown that higher accuracy on ImageNet usually leads to better robustness against different corruptions. Therefore, in this paper, instead of following the traditional research paradigm that investigates new out-of-distribution corruptions or perturbations deep models may encounter, we conduct model debugging in in-distribution data to explore which object attributes a model may be sensitive to. To achieve this goal, we create a toolkit for object editing with controls of backgrounds, sizes, positions, and directions, and create a rigorous benchmark named ImageNet-E(diting) for evaluating the image classifier robustness in terms of object attributes. With our ImageNet-E, we evaluate the performance of current deep learning models, including both convolutional neural networks and vision transformers. We find that most models are quite sensitive to attribute changes. A small change in the background can lead to an average of 9.23% drop on top-1 accuracy. We also evaluate some robust models including both adversarially trained models and other robust trained models and find that some models show worse robustness against attribute changes than vanilla models. Based on these findings, we discover ways to enhance attribute robustness with preprocessing, architecture designs, and training strategies. We hope this work can provide some insights to the community and open up a new avenue for research in robust computer vision. The code and dataset are available at https://github.com/alibaba/easyrobust.