mlsanityData to 7 Oct 2026

Adaptive Testing of Computer Vision Models

arXiv 2212.02774 ↗First posted 6 Dec 2022cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Vision models often fail systematically on groups of data that share common semantic characteristics (e.g., rare objects or unusual scenes), but identifying these failure modes is a challenge. We introduce AdaVision, an interactive process for testing vision models which helps users identify and fix coherent failure modes. Given a natural language description of a coherent group, AdaVision retrieves relevant images from LAION-5B with CLIP. The user then labels a small amount of data for model correctness, which is used in successive retrieval rounds to hill-climb towards high-error regions, refining the group definition. Once a group is saturated, AdaVision uses GPT-3 to suggest new group descriptions for the user to explore. We demonstrate the usefulness and generality of AdaVision in user studies, where users find major bugs in state-of-the-art classification, object detection, and image captioning models. These user-discovered groups have failure rates 2-3x higher than those surfaced by automatic error clustering methods. Finally, finetuning on examples found with AdaVision fixes the discovered bugs when evaluated on unseen examples, without degrading in-distribution accuracy, and while also improving performance on out-of-distribution datasets.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 39 of its 69 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

39 of 69 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2019Natural Adversarial Examplescited by 1,401
27 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2018Model Cards for Model Reportingcited by 1,068
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  9. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  10. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
  11. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
  12. 2019NLNL: Negative Learning for Noisy Labelscited by 151
  13. 2019Does Object Recognition Work for Everyone?cited by 149
  14. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  15. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  16. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  17. 2019Image Counterfactual Sensitivity Analysis for Detecting Unintended Biascited by 74
  18. 2020Towards causal benchmarking of bias in face analysis algorithmscited by 64
  19. 2021The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Modelscited by 63
  20. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 60
  21. 2022Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioningcited by 45
  22. 2022Cycle-Consistent Counterfactuals by Latent Transformationscited by 33
  23. 20213DB: A Framework for Debugging Computer Vision Modelscited by 31
  24. 2022One Ontology to Rule Them All: Corner Case Scenarios for Autonomous Drivingcited by 16
  25. 2020I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptivelycited by 13
  26. 2021Exposing Semantic Segmentation Failures via Maximum Discrepancy Competitioncited by 7
  27. 2022Vision Checklist: Towards Testable Error Analysis of Image Models to Help System Designers Interrogate Model Capabilitiescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 16; 2025: 10; 2026: 3.

39 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  3. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  4. 2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 43
  5. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  6. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  7. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  8. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  9. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
  10. 2023Identification of Systematic Errors of Image Classifiers on Rare Subgroupscited by 17
  11. 2023DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanationscited by 11
  12. 2025HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debuggingcited by 11
27 more
  1. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  2. 2023Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithmscited by 9
  3. 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
  4. 2024Clarify: Improving Model Robustness With Natural Language Correctionscited by 8
  5. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 8
  6. 2024Benchmarking Segmentation Models with Mask-Preserved Attribute Editingcited by 6
  7. 2024Concept-based Analysis of Neural Networks via Vision-Language Modelscited by 6
  8. 2024Visual Data Diagnosis and Debiasing with Concept Graphscited by 6
  9. 2024What could go wrong? Discovering and describing failure modes in computer visioncited by 5
  10. 2025AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Modelscited by 5
  11. 2025VISLIX: An XAI Framework for Validating Vision Models with Slice Discovery and Analysiscited by 5
  12. 2023Identifying Systematic Errors in Object Detectors with the SCROD Pipelinecited by 4
  13. 2024Red-Teaming Segment Anything Modelcited by 4
  14. 2024Learning Where to Edit Vision Transformerscited by 4
  15. 2025Error Slice Discovery via Manifold Compactnesscited by 4
  16. 2025Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifierscited by 4
  17. 2025FailureAtlas:Mapping the Failure Landscape of T2I Models via Active Explorationcited by 2
  18. 2026Adaptive auditing of AI systems with anytime-valid guaranteescited by 2
  19. 2026Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methodscited by 2
  20. 2024Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimatorscited by 1
  21. 2024Interactive Visual Assessment for Text-to-Image Generation Modelscited by 1
  22. 2025Detecting Systematic Weaknesses in Vision Models along Predefined Human-Understandable Dimensionscited by 1
  23. 2024Identification of Fine-grained Systematic Errors via Controlled Scene Generationcited by 0
  24. 2024Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Modelscited by 0
  25. 2024Improving Object Detection by Modifying Synthetic Data with Explainable AIcited by 0
  26. 2025SafeFix: Targeted Model Repair via Controlled Image Generationcited by 0
  27. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0