mlsanityData to 7 Oct 2026

Understanding the Dependence of Perception Model Competency on Regions in an Image

arXiv 2407.10543 ↗First posted 15 Jul 2024cs.CV cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

While deep neural network (DNN)-based perception models are useful for many applications, these models are black boxes and their outputs are not yet well understood. To confidently enable a real-world, decision-making system to utilize such a perception model without human intervention, we must enable the system to reason about the perception model's level of competency and respond appropriately when the model is incompetent. In order for the system to make an intelligent decision about the appropriate action when the model is incompetent, it would be useful for the system to understand why the model is incompetent. We explore five novel methods for identifying regions in the input image contributing to low model competency, which we refer to as image cropping, segment masking, pixel perturbation, competency gradients, and reconstruction loss. We assess the ability of these five methods to identify unfamiliar objects, recognize regions associated with unseen classes, and identify unexplored areas in an environment. We find that the competency gradients and reconstruction loss methods show great promise in identifying regions associated with low model competency, particularly when aspects of the image that are unfamiliar to the perception model are causing this reduction in competency. Both of these methods boast low computation times and high levels of accuracy in detecting image regions that are unfamiliar to the model, allowing them to provide potential utility in decision-making pipelines. The code for reproducing our methods and results is available on GitHub: https://github.com/sarapohland/explainable-competency.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 71 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

42 of 71 references.

References · 42

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2015Variational Inference with Normalizing Flowscited by 2,161
  10. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  12. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
30 more
  1. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  2. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  3. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  4. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  5. 2020Concept Bottleneck Modelscited by 1,022
  6. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  7. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  8. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  9. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  10. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  11. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
  12. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 560
  13. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  14. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  15. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  16. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  17. 2019Counterfactual Visual Explanationscited by 312
  18. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 248
  19. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  20. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  21. 2018Adversarially Learned One-Class Classifier for Novelty Detectioncited by 202
  22. 2019OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representationscited by 199
  23. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  24. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  25. 2018Efficient GAN-Based Anomaly Detectioncited by 122
  26. 2020Gradients as a Measure of Uncertainty in Neural Networkscited by 55
  27. 2018Classification Uncertainty of Deep Neural Networks Based on Gradient Informationcited by 43
  28. 2019Density estimation in representation space to predict model uncertaintycited by 27
  29. 2017Consistent feature attribution for tree ensemblescited by 17
  30. 2019Accurate Layerwise Interpretable Competence Estimationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Mechanistic interpretability for steering vision-language-action modelscited by 30
  2. 2025Explaining Low Perception Model Competency with High-Competency Counterfactualscited by 0
  3. 2026No Single Metric Tells the Whole Story: A Multi-Dimensional Evaluation Framework for Uncertainty Attributionscited by 0