# Explaining Image Classifiers with Multiscale Directional Image Representation

arXiv: https://arxiv.org/abs/2211.12857 · HTML: https://mlsanity.com/paper/2211.12857/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Explaining Recurrent Neural Network Predictions in Sentiment Analysis](https://mlsanity.com/paper/1706.07206/) (2017)
- [A Rate-Distortion Framework for Explaining Neural Network Decisions](https://mlsanity.com/paper/1905.11092/) (2019)
- [Cartoon Explanations of Image Classifiers](https://mlsanity.com/paper/2110.03485/) (2021)
- [Edge, Ridge, and Blob Detection with Symmetric Molecules](https://mlsanity.com/paper/1901.09723/) (2019)
- [Shearlets as Feature Extractor for Semantic Edge Detection: The Model-Based and Data-Driven Realm](https://mlsanity.com/paper/1911.12159/) (2019)
- [In-Distribution Interpretability for Challenging Modalities](https://mlsanity.com/paper/2007.00758/) (2020)
- [Extraction of digital wavefront sets using applied harmonic analysis and deep neural networks](https://mlsanity.com/paper/1901.01388/) (2019)

## Built on by (most cited first, 7 of 7)

- [3VL: Using Trees to Improve Vision-Language Models' Interpretability](https://mlsanity.com/paper/2312.17345/) (2023)
- [One Wave To Explain Them All: A Unifying Perspective On Feature Attribution](https://mlsanity.com/paper/2410.01482/) (2024)
- [On Spectral Properties of Gradient-based Explanation Methods](https://mlsanity.com/paper/2508.10595/) (2025)
- [CHEM: Estimating and Understanding Hallucinations in Deep Learning for Image Processing](https://mlsanity.com/paper/2512.09806/) (2025)
- [On the Interplay of Subset Selection and Informed Graph Neural Networks](https://mlsanity.com/paper/2306.10066/) (2023)
- [Computability of Classification and Deep Learning: From Theoretical Limits to Practical Feasibility through Quantization](https://mlsanity.com/paper/2408.06212/) (2024)
- [Hidden Monotonicity: Explaining Deep Neural Networks via their DC Decomposition](https://mlsanity.com/paper/2601.07700/) (2026)

## Abstract (arXiv)

Image classifiers are known to be difficult to interpret and therefore require explanation methods to understand their decisions. We present ShearletX, a novel mask explanation method for image classifiers based on the shearlet transform -- a multiscale directional image representation. Current mask explanation methods are regularized by smoothness constraints that protect against undesirable fine-grained explanation artifacts. However, the smoothness of a mask limits its ability to separate fine-detail patterns, that are relevant for the classifier, from nearby nuisance patterns, that do not affect the classifier. ShearletX solves this problem by avoiding smoothness regularization all together, replacing it by shearlet sparsity constraints. The resulting explanations consist of a few edges, textures, and smooth parts of the original image, that are the most relevant for the decision of the classifier. To support our method, we propose a mathematical definition for explanation artifacts and an information theoretic score to evaluate the quality of mask explanations. We demonstrate the superiority of ShearletX over previous mask based explanation methods using these new metrics, and present exemplary situations where separating fine-detail patterns allows explaining phenomena that were not explainable before.