# Exploring Interpretable LSTM Neural Networks over Multi-Variable Data

arXiv: https://arxiv.org/abs/1905.12034 · HTML: https://mlsanity.com/paper/1905.12034/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 30 of 42)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Causal Discovery from Temporal Data: An Overview and New Perspectives](https://mlsanity.com/paper/2303.10112/) (2023)
- [WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Values](https://mlsanity.com/paper/2211.06507/) (2022)
- [A Multi-view Multi-task Learning Framework for Multi-variate Time Series Forecasting](https://mlsanity.com/paper/2109.01657/) (2021)
- [Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers](https://mlsanity.com/paper/2206.06131/) (2022)
- [MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset](https://mlsanity.com/paper/2102.06761/) (2021)
- [SSDNet: State Space Decomposition Neural Network for Time Series Forecasting](https://mlsanity.com/paper/2112.10251/) (2021)
- [A Transfer Learning-based State of Charge Estimation for Lithium-Ion Battery at Varying Ambient Temperatures](https://mlsanity.com/paper/2101.03704/) (2021)
- [A Survey of Explainable Artificial Intelligence (XAI) in Financial Time Series Forecasting](https://mlsanity.com/paper/2407.15909/) (2024)
- [Inductive Granger Causal Modeling for Multivariate Time Series](https://mlsanity.com/paper/2102.05298/) (2021)
- [Semi-Supervised Learning and Data Augmentation in Wearable-based Momentary Stress Detection in the Wild](https://mlsanity.com/paper/2202.12935/) (2022)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [MTS2Graph: Interpretable Multivariate Time Series Classification with Temporal Evolving Graphs](https://mlsanity.com/paper/2306.03834/) (2023)
- [Embracing the black box: Heading towards foundation models for causal discovery from time series data](https://mlsanity.com/paper/2402.09305/) (2024)
- [Deep Coupling Network For Multivariate Time Series Forecasting](https://mlsanity.com/paper/2402.15134/) (2024)
- [Mixture of Linear Models Co-supervised by Deep Neural Networks](https://mlsanity.com/paper/2108.04035/) (2021)
- [TreeDRNet:A Robust Deep Model for Long Term Time Series Forecasting](https://mlsanity.com/paper/2206.12106/) (2022)
- [Deep Generative Spatiotemporal Engression for Probabilistic Forecasting of Epidemics](https://mlsanity.com/paper/2603.07108/) (2026)
- [Predicting machine failures from multivariate time series: an industrial case study](https://mlsanity.com/paper/2402.17804/) (2024)
- [WEITS: A Wavelet-enhanced residual framework for interpretable time series forecasting](https://mlsanity.com/paper/2405.10877/) (2024)
- [Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild](https://mlsanity.com/paper/2107.09510/) (2021)
- [NeuroView-RNN: It's About Time](https://mlsanity.com/paper/2202.11811/) (2022)
- [EgPDE-Net: Building Continuous Neural Networks for Time Series Prediction with Exogenous Variables](https://mlsanity.com/paper/2208.01913/) (2022)
- [FV-MgNet: Fully Connected V-cycle MgNet for Interpretable Time Series Forecasting](https://mlsanity.com/paper/2302.00962/) (2023)
- [GARNN: An Interpretable Graph Attentive Recurrent Neural Network for Predicting Blood Glucose Levels via Multivariate Time Series](https://mlsanity.com/paper/2402.16230/) (2024)
- [Interpretable Additive Recurrent Neural Networks For Multivariate Clinical Time Series](https://mlsanity.com/paper/2109.07602/) (2021)
- [The Right Tool for the Job: Open-Source Auditing Tools in Machine Learning](https://mlsanity.com/paper/2206.10613/) (2022)
- [DANLIP: Deep Autoregressive Networks for Locally Interpretable Probabilistic Forecasting](https://mlsanity.com/paper/2301.02332/) (2023)
- [Learning Mixture Structure on Multi-Source Time Series for Probabilistic Forecasting](https://mlsanity.com/paper/2302.11078/) (2023)

## Abstract (arXiv)

For recurrent neural networks trained on time series with target and exogenous variables, in addition to accurate prediction, it is also desired to provide interpretable insights into the data. In this paper, we explore the structure of LSTM recurrent neural networks to learn variable-wise hidden states, with the aim to capture different dynamics in multi-variable time series and distinguish the contribution of variables to the prediction. With these variable-wise hidden states, a mixture attention mechanism is proposed to model the generative process of the target. Then we develop associated training methods to jointly learn network parameters, variable and temporal importance w.r.t the prediction of the target variable. Extensive experiments on real datasets demonstrate enhanced prediction performance by capturing the dynamics of different variables. Meanwhile, we evaluate the interpretation results both qualitatively and quantitatively. It exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variable data.