# Towards a Rigorous Evaluation of Explainability for Multivariate Time Series

arXiv: https://arxiv.org/abs/2104.04075 · HTML: https://mlsanity.com/paper/2104.04075/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Temporal Pattern Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/1809.04206/) (2018)
- [Towards a Rigorous Evaluation of XAI Methods on Time Series](https://mlsanity.com/paper/1909.07082/) (2019)
- [Human Evaluation of Interpretability: The Case of AI-Generated Music Knowledge](https://mlsanity.com/paper/2004.06894/) (2020)

## Built on by (most cited first, 10 of 10)

- [Time is Not Enough: Time-Frequency based Explanation for Time-Series Black-Box Models](https://mlsanity.com/paper/2408.03636/) (2024)
- [TsSHAP: Robust model agnostic feature-based explainability for time series forecasting](https://mlsanity.com/paper/2303.12316/) (2023)
- [EDformer: Embedded Decomposition Transformer for Interpretable Multivariate Time Series Predictions](https://mlsanity.com/paper/2412.12227/) (2024)
- [Explainable Predictive Modeling for Limited Spectral Data](https://mlsanity.com/paper/2202.04527/) (2022)
- [Surrogate Modeling for Explainable Predictive Time Series Corrections](https://mlsanity.com/paper/2412.19897/) (2024)
- [TriShGAN: Enhancing Sparsity and Robustness in Multivariate Time Series Counterfactuals Explanation](https://mlsanity.com/paper/2511.06529/) (2025)
- [Interpretable and Explainable Surrogate Modeling for Simulations: A State-of-the-Art Survey and Perspectives on Explainable AI for Decision-Making](https://mlsanity.com/paper/2604.14240/) (2026)
- [Automated Mobility Context Detection with Inertial Signals](https://mlsanity.com/paper/2205.08409/) (2022)
- [Designing forecasting software for forecast users: Empowering non-experts to create and understand their own forecasts](https://mlsanity.com/paper/2404.14575/) (2024)
- [TSFeatLIME: An Online User Study in Enhancing Explainability in Univariate Time Series Forecasting](https://mlsanity.com/paper/2409.15950/) (2024)

## Abstract (arXiv)

Machine learning-based systems are rapidly gaining popularity and in-line with that there has been a huge research surge in the field of explainability to ensure that machine learning models are reliable, fair, and can be held liable for their decision-making process. Explainable Artificial Intelligence (XAI) methods are typically deployed to debug black-box machine learning models but in comparison to tabular, text, and image data, explainability in time series is still relatively unexplored. The aim of this study was to achieve and evaluate model agnostic explainability in a time series forecasting problem. This work focused on proving a solution for a digital consultancy company aiming to find a data-driven approach in order to understand the effect of their sales related activities on the sales deals closed. The solution involved framing the problem as a time series forecasting problem to predict the sales deals and the explainability was achieved using two novel model agnostic explainability techniques, Local explainable model-agnostic explanations (LIME) and Shapley additive explanations (SHAP) which were evaluated using human evaluation of explainability. The results clearly indicate that the explanations produced by LIME and SHAP greatly helped lay humans in understanding the predictions made by the machine learning model. The presented work can easily be extended to any time