# A Framework to Learn with Interpretation

arXiv: https://arxiv.org/abs/2010.09345 · HTML: https://mlsanity.com/paper/2010.09345/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation](https://mlsanity.com/paper/2108.12204/) (2021)
- [ProtoVAE: A Trustworthy Self-Explainable Prototypical Variational Model](https://mlsanity.com/paper/2210.08151/) (2022)
- [Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMF](https://mlsanity.com/paper/2202.11479/) (2022)
- [TrajPAC: Towards Robustness Verification of Pedestrian Trajectory Prediction Models](https://mlsanity.com/paper/2308.05985/) (2023)
- [Interpretability-Aware Vision Transformer](https://mlsanity.com/paper/2309.08035/) (2023)
- [Focal Modulation Networks for Interpretable Sound Classification](https://mlsanity.com/paper/2402.02754/) (2024)
- [Posthoc Interpretation via Quantization](https://mlsanity.com/paper/2303.12659/) (2023)
- [Adversarial Attacks on the Interpretation of Neuron Activation Maximization](https://mlsanity.com/paper/2306.07397/) (2023)
- [A Unified Study of Machine Learning Explanation Evaluation Metrics](https://mlsanity.com/paper/2203.14265/) (2022)
- [Prototypical Self-Explainable Models Without Re-training](https://mlsanity.com/paper/2312.07822/) (2023)
- [Tackling Interpretability in Audio Classification Networks with Non-negative Matrix Factorization](https://mlsanity.com/paper/2305.07132/) (2023)
- [The Gaussian Discriminant Variational Autoencoder (GdVAE): A Self-Explainable Model with Counterfactual Explanations](https://mlsanity.com/paper/2409.12952/) (2024)
- [LMAC-TD: Producing Time Domain Explanations for Audio Classifiers](https://mlsanity.com/paper/2409.08655/) (2024)
- [Towards credible visual model interpretation with path attribution](https://mlsanity.com/paper/2305.14395/) (2023)
- [This Probably Looks Exactly Like That: An Invertible Prototypical Network](https://mlsanity.com/paper/2407.12200/) (2024)
- [Restyling Unsupervised Concept Based Interpretable Networks with Generative Models](https://mlsanity.com/paper/2407.01331/) (2024)
- [Statistical process monitoring of artificial neural networks](https://mlsanity.com/paper/2209.07436/) (2022)
- [Hidden in Plain Sight -- Class Competition Focuses Attribution Maps](https://mlsanity.com/paper/2503.07346/) (2025)
- [BELLA: Black box model Explanations by Local Linear Approximations](https://mlsanity.com/paper/2305.11311/) (2023)
- [Abnormal component analysis](https://mlsanity.com/paper/2312.16139/) (2023)
- [Learning local discrete features in explainable-by-design convolutional neural networks](https://mlsanity.com/paper/2411.00139/) (2024)
- [TimeSAE: Causal Sparse Decoding for Faithful Explanations of Black-Box Time Series Models](https://mlsanity.com/paper/2601.09776/) (2026)
- [Post-hoc Self-explanation of CNNs](https://mlsanity.com/paper/2603.28466/) (2026)

## Abstract (arXiv)

To tackle interpretability in deep learning, we present a novel framework to jointly learn a predictive model and its associated interpretation model. The interpreter provides both local and global interpretability about the predictive model in terms of human-understandable high level attribute functions, with minimal loss of accuracy. This is achieved by a dedicated architecture and well chosen regularization penalties. We seek for a small-size dictionary of high level attribute functions that take as inputs the outputs of selected hidden layers and whose outputs feed a linear classifier. We impose strong conciseness on the activation of attributes with an entropy-based criterion while enforcing fidelity to both inputs and outputs of the predictive model. A detailed pipeline to visualize the learnt features is also developed. Moreover, besides generating interpretable models by design, our approach can be specialized to provide post-hoc interpretations for a pre-trained neural network. We validate our approach against several state-of-the-art methods on multiple datasets and show its efficacy on both kinds of tasks.