# On the Veracity of Local, Model-agnostic Explanations in Audio Classification: Targeted Investigations with Adversarial Examples

arXiv: https://arxiv.org/abs/2107.09045 · HTML: https://mlsanity.com/paper/2107.09045/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 12 of 12)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Audio Adversarial Examples: Targeted Attacks on Speech-to-Text](https://mlsanity.com/paper/1801.01944/) (2018)
- ["How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations](https://mlsanity.com/paper/1911.06473/) (2019)
- [audioLIME: Listenable Explanations Using Source Separation](https://mlsanity.com/paper/2008.00582/) (2020)
- [Reliable Local Explanations for Machine Listening](https://mlsanity.com/paper/2005.07788/) (2020)
- [GAN-based Generation and Automatic Selection of Explanations for Neural Networks](https://mlsanity.com/paper/1904.09533/) (2019)
- [Towards Musically Meaningful Explanations Using Source Separation](https://mlsanity.com/paper/2009.02051/) (2020)
- [Two-level Explanations in Music Emotion Recognition](https://mlsanity.com/paper/1905.11760/) (2019)
- [Recovering Localized Adversarial Attacks](https://mlsanity.com/paper/1910.09239/) (2019)

## Built on by (most cited first, 3 of 3)

- [Explainability in Music Recommender Systems](https://mlsanity.com/paper/2201.10528/) (2022)
- [Concept-Based Techniques for "Musicologist-friendly" Explanations in a Deep Music Classifier](https://mlsanity.com/paper/2208.12485/) (2022)
- [Predicting Music Hierarchies with a Graph-Based Neural Decoder](https://mlsanity.com/paper/2306.16955/) (2023)

## Abstract (arXiv)

Local explanation methods such as LIME have become popular in MIR as tools for generating post-hoc, model-agnostic explanations of a model's classification decisions. The basic idea is to identify a small set of human-understandable features of the classified example that are most influential on the classifier's prediction. These are then presented as an explanation. Evaluation of such explanations in publications often resorts to accepting what matches the expectation of a human without actually being able to verify if what the explanation shows is what really caused the model's prediction. This paper reports on targeted investigations where we try to get more insight into the actual veracity of LIME's explanations in an audio classification task. We deliberately design adversarial examples for the classifier, in a way that gives us knowledge about which parts of the input are potentially responsible for the model's (wrong) prediction. Asking LIME to explain the predictions for these adversaries permits us to study whether local explanations do indeed detect these regions of interest. We also look at whether LIME is more successful in finding perturbations that are more prominent and easily noticeable for a human. Our results suggest that LIME does not necessarily manage to identify the most relevant input features and hence it remains unclear whether explanations are useful or even misleading.