mlsanityData to 7 Oct 2026

On Pitfalls of textitRemOve-And-Retrain: Data Processing Inequality Perspective

arXiv 2304.13836 ↗First posted 26 Apr 2023cs.LG cs.AI cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

The RemOve-And-Retrain (ROAR) benchmark is widely used to evaluate feature attribution methods, yet its validity remains underexplored from an information-theoretic perspective. We show that model- and data-agnostic post-processing of attribution maps (transformations that, by the data processing inequality, cannot add information about the decision function) can often improve ROAR scores. This means that an improved ROAR ranking is not, by itself, evidence that an attribution map carries more information about the model. We trace this failure mode to a bias toward spatially blurry masks. Experiments on CIFAR-10, SVHN, and CUB-200 show a consistent association between blurriness and ROAR performance, a pattern that also appears in the ROAD variant. We provide guidelines for more cautious removal-based benchmarking, with implications for validating mechanistic understanding of neural network internals.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 66 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

35 of 66 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  8. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  9. 2018Sanity Checks for Saliency Mapscited by 1,149
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  12. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
23 more
  1. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  2. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  3. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  4. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  5. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
  6. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  7. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  8. 2020Evaluating and Aggregating Feature-based Model Explanationscited by 185
  9. 2020Debugging Tests for Model Explanationscited by 146
  10. 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 141
  11. 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
  12. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 100
  13. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  14. 2020Generative causal explanations of black-box classifierscited by 41
  15. 2018Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Valuescited by 40
  16. 2019Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Mapscited by 34
  17. 2021Zorro: Valid, Sparse, and Stable Explanations in Graph Neural Networkscited by 34
  18. 2021Neural Response Interpretation through the Lens of Critical Pathwayscited by 32
  19. 2019Bridging Adversarial Robustness and Gradient Interpretabilitycited by 22
  20. 2021On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluationcited by 12
  21. 2020Adversarial Infidelity Learning for Model Interpretationcited by 9
  22. 2020Learning Propagation Rules for Attribution Map Generationcited by 8
  23. 2022Rethinking Efficiency and Redundancy in Training Large-scale Graphscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Geometric Remove-and-Retrain (GOAR): Coordinate-Invariant eXplainable AI Assessmentcited by 3
  2. 2024Guided AbsoluteGrad: Magnitude of Gradients Matters to Explanation's Localization and Saliencycited by 0