# Interpretation of Prediction Models Using the Input Gradient

arXiv: https://arxiv.org/abs/1611.07634 · HTML: https://mlsanity.com/paper/1611.07634/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2016.

## Built on by (most cited first, 24 of 24)

- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [Learning by Self-Explaining](https://mlsanity.com/paper/2309.08395/) (2023)
- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing](https://mlsanity.com/paper/2205.04559/) (2022)
- [Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks](https://mlsanity.com/paper/2104.04488/) (2021)
- [Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution](https://mlsanity.com/paper/2106.01518/) (2021)
- [A Typology for Exploring the Mitigation of Shortcut Behavior](https://mlsanity.com/paper/2203.03668/) (2022)
- [PromptExp: Multi-granularity Prompt Explanation of Large Language Models](https://mlsanity.com/paper/2410.13073/) (2024)
- [Deep neural networks with controlled variable selection for the identification of putative causal genetic variants](https://mlsanity.com/paper/2109.14719/) (2021)
- [Faster Verified Explanations for Neural Networks](https://mlsanity.com/paper/2512.00164/) (2025)
- [Evaluating the Reliability of Self-Explanations in Large Language Models](https://mlsanity.com/paper/2407.14487/) (2024)
- [Making Document-Level Information Extraction Right for the Right Reasons](https://mlsanity.com/paper/2110.07686/) (2021)
- [Identifying the Source of Vulnerability in Explanation Discrepancy: A Case Study in Neural Text Classification](https://mlsanity.com/paper/2212.05327/) (2022)
- [One Explanation Does Not Fit XIL](https://mlsanity.com/paper/2304.07136/) (2023)
- [The Conformer Encoder May Reverse the Time Dimension](https://mlsanity.com/paper/2410.00680/) (2024)
- [Look Who's Talking: Interpretable Machine Learning for Assessing Italian SMEs Credit Default](https://mlsanity.com/paper/2108.13914/) (2021)
- [Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking](https://mlsanity.com/paper/2203.14474/) (2022)
- [Model interpretation using improved local regression with variable importance](https://mlsanity.com/paper/2209.05371/) (2022)
- [Abstract Interpretation-Based Feature Importance for SVMs](https://mlsanity.com/paper/2210.12456/) (2022)
- [Regression Trees Know Calculus](https://mlsanity.com/paper/2405.13846/) (2024)
- [CatNet: Controlling the False Discovery Rate in LSTM with SHAP Feature Importance and Gaussian Mirrors](https://mlsanity.com/paper/2411.16666/) (2024)
- [MASE: Interpretable NLP Models via Model-Agnostic Saliency Estimation](https://mlsanity.com/paper/2512.04386/) (2025)
- [Real-Time Visual Attribution Streaming in Thinking Model](https://mlsanity.com/paper/2604.16587/) (2026)
- [Model-Agnostic FDR Control via Group Gaussian Mirror and Permutation SHAP](https://mlsanity.com/paper/2608.00989/) (2026)

## Abstract (arXiv)

State of the art machine learning algorithms are highly optimized to provide the optimal prediction possible, naturally resulting in complex models. While these models often outperform simpler more interpretable models by order of magnitudes, in terms of understanding the way the model functions, we are often facing a "black box". In this paper we suggest a simple method to interpret the behavior of any predictive model, both for regression and classification. Given a particular model, the information required to interpret it can be obtained by studying the partial derivatives of the model with respect to the input. We exemplify this insight by interpreting convolutional and multi-layer neural networks in the field of natural language processing.