mlsanityData to 7 Oct 2026

Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation

arXiv 2212.04629 ↗First posted 9 Dec 2022cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

We investigate whether three types of post hoc model explanations--feature attribution, concept activation, and training point ranking--are effective for detecting a model's reliance on spurious signals in the training data. Specifically, we consider the scenario where the spurious signal to be detected is unknown, at test-time, to the user of the explanation method. We design an empirical methodology that uses semi-synthetic datasets along with pre-specified spurious artifacts to obtain models that verifiably rely on these spurious training signals. We then provide a suite of metrics that assess an explanation method's reliability for spurious signal detection under various conditions. We find that the post hoc explanation methods tested are ineffective when the spurious artifact is unknown at test-time especially for non-visible artifacts like a background blur. Further, we find that feature attribution methods are susceptible to erroneously indicating dependence on spurious signals even when the model being explained does not rely on spurious artifacts. This finding casts doubt on the utility of these approaches, in the hands of a practitioner, for detecting a model's reliance on spurious signals.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 53 of its 72 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

53 of 72 references.

References · 53

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  7. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  10. 2018Sanity Checks for Saliency Mapscited by 1,149
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  12. 2020Concept Bottleneck Modelscited by 1,022
38 more
  1. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  2. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  3. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  4. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  5. 2017Interpretation of Neural Networks is Fragilecited by 443
  6. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
  7. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  8. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  9. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  10. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  11. 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 293
  12. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  13. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  14. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  15. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  16. 2019Explanations can be manipulated and geometry is to blamecited by 215
  17. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  18. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  19. 2020Debugging Tests for Model Explanationscited by 146
  20. 2019Interpretations are useful: penalizing explanations to align neural networks with prior knowledgecited by 144
  21. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  22. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  23. 2019Sanity Checks for Saliency Metricscited by 119
  24. 2019Fooling Neural Network Interpretations via Adversarial Model Manipulationcited by 112
  25. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  26. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  27. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  28. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  29. 2020Fairwashing Explanations with Off-Manifold Detergentcited by 71
  30. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  31. 2018A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizationscited by 59
  32. 2020Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionatelycited by 52
  33. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  34. 2020Now You See Me (CME): Concept-based Model Extractioncited by 51
  35. 2018Deep Learning Predicts Hip Fracture using Confounding Patient and Healthcare Variablescited by 50
  36. 2020How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labelscited by 39
  37. 2021Finding and Fixing Spurious Patterns with Explanationscited by 38
  38. 2020Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Predictioncited by 36

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 23; 2024: 19; 2025: 9; 2026: 7.

59 papers.

Sort:

By area:

  1. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  2. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  3. 2023Concept-based Explainable Artificial Intelligence: A Surveycited by 97
  4. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  5. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  6. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  7. 2023Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generationcited by 41
  8. 2024Learned feature representations are biased by complexity, learning order, position, and morecited by 22
  9. 2023Studying How to Efficiently and Effectively Guide Models with Explanationscited by 16
  10. 2023Right for the Wrong Reason: Can Interpretable ML Techniques Detect Spurious Correlations?cited by 16
  11. 2022On the Relationship Between Explanation and Prediction: A Causal Viewcited by 14
  12. 2023Stability Guarantees for Feature Attributions with Multiplicative Smoothingcited by 14
38 more
  1. 2023Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UKcited by 13
  2. 2024B-cosification: Transforming Deep Neural Networks to be Inherently Interpretablecited by 12
  3. 2023DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanationscited by 11
  4. 2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
  5. 2024Interpretability Needs a New Paradigmcited by 10
  6. 2024Regulating Model Reliance on Non-Robust Features by Smoothing Input Marginal Densitycited by 10
  7. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  8. 2023Preemptively Pruning Clever-Hans Strategies in Deep Neural Networkscited by 9
  9. 2023Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithmscited by 9
  10. 2025Probabilistic Stability Guarantees for Feature Attributionscited by 9
  11. 2023Feature Interactions Reveal Linguistic Structure in Language Modelscited by 8
  12. 2024Rethinking Explainable Machine Learning as Applied Statisticscited by 8
  13. 2024A survey on Concept-based Approaches For Model Improvementcited by 8
  14. 2023Error Discovery by Clustering Influence Embeddingscited by 7
  15. 2024Detecting music deepfakes is easy but actually hardcited by 7
  16. 2024Understanding Disparities in Post Hoc Machine Learning Explanationcited by 6
  17. 2024How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency mapscited by 6
  18. 2024Interpretable Prognostics with Concept Bottleneck Modelscited by 6
  19. 2025Efficient Unsupervised Shortcut Learning Detection and Mitigation in Transformerscited by 6
  20. 2023Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goalscited by 4
  21. 2023Two-Stage Holistic and Contrastive Explanation of Image Classificationcited by 4
  22. 2023Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Trainingcited by 4
  23. 2024Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentationcited by 4
  24. 2023Interpreting Vision and Language Generative Models with Semantic Visual Priorscited by 2
  25. 2023Looking deeper into interpretable deep learning in neuroimaging: a comprehensive surveycited by 2
  26. 2025Prototype-Based Multiple Instance Learning for Gigapixel Whole Slide Image Classificationcited by 2
  27. 2023Iterative Partial Fulfillment of Counterfactual Explanations: Benefits and Riskscited by 1
  28. 2023UFO: A unified method for controlling Understandability and Faithfulness Objectives in concept-based explanations for CNNscited by 1
  29. 2023Detecting Spurious Correlations via Robust Visual Concepts in Real and AI-Generated Image Classificationcited by 1
  30. 2024Prospector Heads: Generalized Feature Attribution for Large Models & Datacited by 1
  31. 2024Towards Understanding Sensitive and Decisive Patterns in Explainable AI: A Case Study of Model Interpretation in Geometric Deep Learningcited by 1
  32. 2024DEPICT: Diffusion-Enabled Permutation Importance for Image Classification Taskscited by 1
  33. 2024In Defence of Post-hoc Explainabilitycited by 1
  34. 2025An interpretable approach to automating the assessment of biofouling in video footagecited by 1
  35. 2025In defence of post-hoc explanations in medical AIcited by 1
  36. 2024Probabilistic Shapley Value Modeling and Inferencecited by 0
  37. 2025TLXML: Task-Level Explanation of Meta-Learning via Influence Functionscited by 0
  38. 2025Are We Merely Justifying Results ex Post Facto? Quantifying Explanatory Inversion in Post-Hoc Model Explanationscited by 0

50 most cited of 59 All 59 (JSON)