# Failure Modes of Time Series Interpretability Algorithms for Critical Care Applications and Potential Solutions

arXiv: https://arxiv.org/abs/2506.19035 · HTML: https://mlsanity.com/paper/2506.19035/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [Interpretability of machine learning based prediction models in healthcare](https://mlsanity.com/paper/2002.08596/) (2020)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Sequential Integrated Gradients: a simple but effective method for explaining language models](https://mlsanity.com/paper/2305.15853/) (2023)
- [HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data](https://mlsanity.com/paper/2111.08536/) (2021)
- [Clinical Intervention Prediction and Understanding using Deep Networks](https://mlsanity.com/paper/1705.08498/) (2017)
- [Learning Perturbations to Explain Time Series Predictions](https://mlsanity.com/paper/2305.18840/) (2023)
- [Time Interpret: a Unified Model Interpretability Library for Time Series](https://mlsanity.com/paper/2306.02968/) (2023)
- [Impact of Accuracy on Model Interpretations](https://mlsanity.com/paper/2011.09903/) (2020)

## Built on by (most cited first, 1 of 1)

- [Data-Driven Discovery of Feature Groups in Clinical Time Series](https://mlsanity.com/paper/2511.08260/) (2025)

## Abstract (arXiv)

Interpretability plays a vital role in aligning and deploying deep learning models in critical care, especially in constantly evolving conditions that influence patient survival. However, common interpretability algorithms face unique challenges when applied to dynamic prediction tasks, where patient trajectories evolve over time. Gradient, Occlusion, and Permutation-based methods often struggle with time-varying target dependency and temporal smoothness. This work systematically analyzes these failure modes and supports learnable mask-based interpretability frameworks as alternatives, which can incorporate temporal continuity and label consistency constraints to learn feature importance over time. Here, we propose that learnable mask-based approaches for dynamic timeseries prediction problems provide more reliable and consistent interpretations for applications in critical care and similar domains.