# Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1910.12370 · HTML: https://mlsanity.com/paper/1910.12370/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2019.

## Built on by (most cited first, 20 of 20)

- [Salient ImageNet: How to discover spurious features in Deep Learning?](https://mlsanity.com/paper/2110.04301/) (2021)
- [Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement](https://mlsanity.com/paper/2301.03028/) (2023)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Improving Deep Learning Interpretability by Saliency Guided Training](https://mlsanity.com/paper/2111.14338/) (2021)
- [WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Values](https://mlsanity.com/paper/2211.06507/) (2022)
- [BolT: Fused Window Transformers for fMRI Time Series Analysis](https://mlsanity.com/paper/2205.11578/) (2022)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [Scrutinizing XAI using linear ground-truth data with suppressor variables](https://mlsanity.com/paper/2111.07473/) (2021)
- [Core Risk Minimization using Salient ImageNet](https://mlsanity.com/paper/2203.15566/) (2022)
- [Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification](https://mlsanity.com/paper/2306.12150/) (2023)
- [Feature Importance Explanations for Temporal Black-Box Models](https://mlsanity.com/paper/2102.11934/) (2021)
- [GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations](https://mlsanity.com/paper/2406.11547/) (2024)
- [Explainable AI needs formalization](https://mlsanity.com/paper/2409.14590/) (2024)
- [Data-Centric Debugging: mitigating model failures via targeted data collection](https://mlsanity.com/paper/2211.09859/) (2022)
- [Applications of interpretable deep learning in neuroimaging: a comprehensive review](https://mlsanity.com/paper/2406.17792/) (2024)
- [Finding Short Signals in Long Irregular Time Series with Continuous-Time Attention Policy Networks](https://mlsanity.com/paper/2302.04052/) (2023)
- [SCAAT: Improving Neural Network Interpretability via Saliency Constrained Adaptive Adversarial Training](https://mlsanity.com/paper/2311.05143/) (2023)
- [Hiding Backdoors within Event Sequence Data via Poisoning Attacks](https://mlsanity.com/paper/2308.10201/) (2023)
- [Evaluating Scenario-based Decision-making for Interactive Autonomous Driving Using Rational Criteria: A Survey](https://mlsanity.com/paper/2501.01886/) (2025)
- [Robust representations of oil wells' intervals via sparse attention mechanism](https://mlsanity.com/paper/2212.14246/) (2022)

## Abstract (arXiv)

Recent efforts to improve the interpretability of deep neural networks use saliency to characterize the importance of input features to predictions made by models. Work on interpretability using saliency-based methods on Recurrent Neural Networks (RNNs) has mostly targeted language tasks, and their applicability to time series data is less understood. In this work we analyze saliency-based methods for RNNs, both classical and gated cell architectures. We show that RNN saliency vanishes over time, biasing detection of salient features only to later time steps and are, therefore, incapable of reliably detecting important features at arbitrary time intervals. To address this vanishing saliency problem, we propose a novel RNN cell structure (input-cell attention), which can extend any RNN cell architecture. At each time step, instead of only looking at the current input vector, input-cell attention uses a fixed-size matrix embedding, each row of the matrix attending to different inputs from current or previous time steps. Using synthetic data, we show that the saliency map produced by the input-cell attention RNN is able to faithfully detect important features regardless of their occurrence in time. We also apply the input-cell attention RNN on a neuroscience task analyzing functional Magnetic Resonance Imaging (fMRI) data for human subjects performing a variety of tasks. In this case, we use saliency to characterize brain regions (input features) for which activity is important to distinguish between tasks. We show that standard RNN architectures are only capable of detecting important brain regions in the last few time steps of the fMRI data, while the input-cell attention model is able to detect important brain region activity across time without latter time step biases.