# NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks

arXiv: https://arxiv.org/abs/1909.04200 · HTML: https://mlsanity.com/paper/1909.04200/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 16 of 16)

- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [SurvNAM: The machine learning survival model explanation](https://mlsanity.com/paper/2104.08903/) (2021)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [Interpretability Needs a New Paradigm](https://mlsanity.com/paper/2405.05386/) (2024)
- [Which LIME should I trust? Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2503.24365/) (2025)
- [Cross-Model Consensus of Explanations and Beyond for Image Classification Models: An Empirical Study](https://mlsanity.com/paper/2109.00707/) (2021)
- [A Survey of Computer Vision Technologies In Urban and Controlled-environment Agriculture](https://mlsanity.com/paper/2210.11318/) (2022)
- [SurvBeX: An explanation method of the machine learning survival models based on the Beran estimator](https://mlsanity.com/paper/2308.03730/) (2023)
- [Towards Understanding the Effectiveness of Attention Mechanism](https://mlsanity.com/paper/2106.15067/) (2021)
- [LIMEcraft: Handcrafted superpixel selection and inspection for Visual eXplanations](https://mlsanity.com/paper/2111.08094/) (2021)
- [Measuring Feature Dependency of Neural Networks by Collapsing Feature Dimensions in the Data Manifold](https://mlsanity.com/paper/2404.12341/) (2024)
- [On original and latent space connectivity in deep neural networks](https://mlsanity.com/paper/2311.06816/) (2023)
- [SurvBeNIM: The Beran-Based Neural Importance Model for Explaining the Survival Models](https://mlsanity.com/paper/2312.06638/) (2023)
- [Integrated feature analysis for deep learning interpretation and class activation maps](https://mlsanity.com/paper/2407.01142/) (2024)
- [Fundamental Limitation in Explaining AI](https://mlsanity.com/paper/2605.24727/) (2026)

## Abstract (arXiv)

The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each approximating a large neural network on a small region of the data manifold and SP-LIME aggregates the local models to form a global interpretation. Extending this line of research, we propose a simple yet effective method, NormLIME for aggregating local models into global and class-specific interpretations. A human user study strongly favored class-specific interpretations created by NormLIME to other feature importance metrics. Numerical experiments confirm that NormLIME is effective at recognizing important features.