# A Protocol for Continual Explanation of SHAP

arXiv: https://arxiv.org/abs/2306.07218 · HTML: https://mlsanity.com/paper/2306.07218/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 8 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 8 of 8)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting](https://mlsanity.com/paper/2010.01528/) (2020)
- [Continual Learning for Recurrent Neural Networks: an Empirical Evaluation](https://mlsanity.com/paper/2103.07492/) (2021)
- [Avalanche: A PyTorch Library for Deep Continual Learning](https://mlsanity.com/paper/2302.01766/) (2023)
- [ICICLE: Interpretable Class Incremental Continual Learning](https://mlsanity.com/paper/2303.07811/) (2023)
- [Continual Learning with Echo State Networks](https://mlsanity.com/paper/2105.07674/) (2021)

## Built on by (most cited first, 2 of 2)

- [CIP-Net: Continual Interpretable Prototype-based Network](https://mlsanity.com/paper/2512.07981/) (2025)
- [Audit Before You Update: Recording Artifacts and the Limits of Bounded Model Updates in MIMIC-IV ICU Prediction](https://mlsanity.com/paper/2607.19020/) (2026)

## Abstract (arXiv)

Continual Learning trains models on a stream of data, with the aim of learning new information without forgetting previous knowledge. Given the dynamic nature of such environments, explaining the predictions of these models can be challenging. We study the behavior of SHAP values explanations in Continual Learning and propose an evaluation protocol to robustly assess the change of explanations in Class-Incremental scenarios. We observed that, while Replay strategies enforce the stability of SHAP values in feedforward/convolutional models, they are not able to do the same with fully-trained recurrent models. We show that alternative recurrent approaches, like randomized recurrent models, are more effective in keeping the explanations stable over time.