mlsanityData to 14 Oct 2026

Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation

arXiv 2302.07865 ↗First posted 15 Feb 2023cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Distribution shift is a major source of failure for machine learning models. However, evaluating model reliability under distribution shift can be challenging, especially since it may be difficult to acquire counterfactual examples that exhibit a specified shift. In this work, we introduce the notion of a dataset interface: a framework that, given an input dataset and a user-specified shift, returns instances from that input distribution that exhibit the desired shift. We study a number of natural implementations for such an interface, and find that they often introduce confounding shifts that complicate model evaluation. Motivated by this, we propose a dataset interface implementation that leverages Textual Inversion to tailor generation to the input distribution. We then demonstrate how applying this dataset interface to the ImageNet dataset enables studying model behavior across a diverse array of distribution shifts, including variations in background, lighting, and attributes of the objects. Code available at https://github.com/MadryLab/dataset-interfaces.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

30 of 40 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
18 more
  1. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  2. 2018Recognition in Terra Incognitacited by 658
  3. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  4. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  5. 2017Functional Map of the Worldcited by 338
  6. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  7. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  8. 2018Benchmarking Neural Network Robustness to Common Corruptions and Surface Variationscited by 99
  9. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  10. 2022Diffusion Models for Counterfactual Explanationscited by 79
  11. 2022Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioningcited by 45
  12. 2022Adaptive Testing of Computer Vision Modelscited by 39
  13. 20213DB: A Framework for Debugging Computer Vision Modelscited by 31
  14. 2020Identifying Statistical Bias in Dataset Replicationcited by 24
  15. 2018SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applicationscited by 12
  16. 2019Identifying Model Weakness with Adversarial Examinercited by 12
  17. 2022Invariant Learning via Diffusion Dreamed Distribution Shiftscited by 10
  18. 2019Towards Analyzing Semantic Robustness of Deep Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 15; 2025: 4; 2026: 2.

35 papers.

Sort:

By area:

  1. 2023EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methodscited by 54
  2. 2023Describing Differences in Image Sets with Natural Languagecited by 53
  3. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  4. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 43
  5. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  6. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  7. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  8. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matchingcited by 36
  9. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33
  10. 2023Text-to-Image Models for Counterfactual Explanations: a Black-Box Approachcited by 28
  11. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27
  12. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
23 more
  1. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  2. 2023Identification of Systematic Errors of Image Classifiers on Rare Subgroupscited by 17
  3. 2024Global Counterfactual Directionscited by 15
  4. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  5. 2023DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanationscited by 11
  6. 2023Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithmscited by 9
  7. 2023PRIME: Prioritizing Interpretability in Failure Mode Extractioncited by 9
  8. 2024Clarify: Improving Model Robustness With Natural Language Correctionscited by 8
  9. 2024Benchmarking Segmentation Models with Mask-Preserved Attribute Editingcited by 6
  10. 2023ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlationscited by 5
  11. 2024What could go wrong? Discovering and describing failure modes in computer visioncited by 5
  12. 2024Unsupervised Model Diagnosiscited by 4
  13. 2024Learning Where to Edit Vision Transformerscited by 4
  14. 2025Diffusion on Graph: Augmentation of Graph Structure for Node Classificationcited by 4
  15. 2024Evaluating Model Bias Requires Characterizing its Mistakescited by 3
  16. 2025DILLEMA: Diffusion and Large Language Models for Multi-Modal Augmentationcited by 3
  17. 2024Difficulty Controlled Diffusion Model for Synthesizing Effective Training Datacited by 2
  18. 2025CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shiftscited by 2
  19. 2026Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editingcited by 2
  20. 2024Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimatorscited by 1
  21. 2024Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Modelscited by 0
  22. 2025SafeFix: Targeted Model Repair via Controlled Image Generationcited by 0
  23. 2026Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Languagecited by 0