# Feature Attribution-Based Explainability Analysis of Deep Learning Models in Predictive Process Monitoring

arXiv: https://arxiv.org/abs/2607.17783 · HTML: https://mlsanity.com/paper/2607.17783/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Predictive Business Process Monitoring with LSTM Neural Networks](https://mlsanity.com/paper/1612.02130/) (2016)
- [TimeSHAP: Explaining Recurrent Models through Sequence Perturbations](https://mlsanity.com/paper/2012.00073/) (2020)
- [Explaining Time Series Predictions with Dynamic Masks](https://mlsanity.com/paper/2106.05303/) (2021)
- [Evaluation of post-hoc interpretability methods in time-series classification](https://mlsanity.com/paper/2202.05656/) (2022)
- [TS-MULE: Local Interpretable Model-Agnostic Explanations for Time Series Forecast Models](https://mlsanity.com/paper/2109.08438/) (2021)
- [Explainable Predictive Process Monitoring](https://mlsanity.com/paper/2008.01807/) (2020)
- [WindowSHAP: An Efficient Framework for Explaining Time-series Classifiers based on Shapley Values](https://mlsanity.com/paper/2211.06507/) (2022)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [Process Outcome Prediction: CNN vs. LSTM (with Attention)](https://mlsanity.com/paper/2104.06934/) (2021)
- [Automatic Change-Point Detection in Time Series via Deep Learning](https://mlsanity.com/paper/2211.03860/) (2022)
- [An Entropic Relevance Measure for Stochastic Conformance Checking in Process Mining](https://mlsanity.com/paper/2007.09310/) (2020)
- [Class-Specific Explainability for Deep Time Series Classifiers](https://mlsanity.com/paper/2210.05411/) (2022)
- [Class-Dependent Perturbation Effects in Evaluating Time Series Attributions](https://mlsanity.com/paper/2502.17022/) (2025)
- [Generating Feasible and Plausible Counterfactual Explanations for Outcome Prediction of Business Processes](https://mlsanity.com/paper/2403.09232/) (2024)
- [SimBank: from Simulation to Solution in Prescriptive Process Monitoring](https://mlsanity.com/paper/2506.14772/) (2025)
- [Guiding the generation of counterfactual explanations through temporal background knowledge for Predictive Process Monitoring](https://mlsanity.com/paper/2403.11642/) (2024)
- [Model-driven Stochastic Trace Clustering](https://mlsanity.com/paper/2506.23776/) (2025)

## Abstract (arXiv)

Predictive process monitoring supports the optimization and control of operational business processes by forecasting the future state or outcome of ongoing cases. While deep neural networks have achieved strong performance for these tasks by modeling sequential dependencies in event logs, their black-box nature limits trust and practical adoption. Feature attribution methods are often used to address this, but applying them directly poses a dilemma: event-level attributions impose high computational complexity for long traces, while explanations based on aggregated trace representations often fail to capture the underlying control-flow dynamics. To address this issue, we propose a local post-hoc explainability method for deep neural networks in outcome prediction. The method relies on a control-flow-aware segmentation algorithm that partitions a trace into meaningful segments and supports the computation of segment-level SHAP explanations. This makes it possible to identify which parts of a trace influence a prediction and which change points steer the case toward the predicted outcome. We assess the proposed segmentation method on a synthetic dataset with known process logic, where meaningful change points can be explicitly verified, and we demonstrate its usefulness on real-world event logs from a loan application process and an administrative process of a Dutch municipality.