# Explainability in Process Outcome Prediction: Guidelines to Obtain Interpretable and Faithful Models

arXiv: https://arxiv.org/abs/2203.16073 · HTML: https://mlsanity.com/paper/2203.16073/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Predictive Business Process Monitoring with LSTM Neural Networks](https://mlsanity.com/paper/1612.02130/) (2016)
- [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020)
- [On quantitative aspects of model interpretability](https://mlsanity.com/paper/2007.07584/) (2020)
- [Predicting Process Behaviour using Deep Learning](https://mlsanity.com/paper/1612.04600/) (2016)
- [Interpretable & Explorable Approximations of Black Box Models](https://mlsanity.com/paper/1707.01154/) (2017)
- [A systematic review and taxonomy of explanations in decision support and recommender systems](https://mlsanity.com/paper/2006.08672/) (2020)
- [Predictive Monitoring of Business Processes](https://mlsanity.com/paper/1312.4874/) (2013)
- [Outcome-Oriented Predictive Process Monitoring: Review and Benchmark](https://mlsanity.com/paper/1707.06766/) (2017)
- [Generalized Linear Rule Models](https://mlsanity.com/paper/1906.01761/) (2019)
- [ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs](https://mlsanity.com/paper/1512.05193/) (2015)
- [Explainable Predictive Process Monitoring](https://mlsanity.com/paper/2008.01807/) (2020)
- [Clustering-Based Predictive Process Monitoring](https://mlsanity.com/paper/1506.01428/) (2015)
- [Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretability](https://mlsanity.com/paper/1904.03867/) (2019)
- [Explainable Artificial Intelligence for Process Mining: A General Overview and Application of a Novel Local Explanation Approach for Predictive Process Monitoring](https://mlsanity.com/paper/2009.02098/) (2020)
- [Predictive and Causal Implications of using Shapley Value for Model Interpretation](https://mlsanity.com/paper/2008.05052/) (2020)
- [Towards Quantification of Explainability in Explainable Artificial Intelligence Methods](https://mlsanity.com/paper/1911.10104/) (2019)
- [Process Outcome Prediction: CNN vs. LSTM (with Attention)](https://mlsanity.com/paper/2104.06934/) (2021)

## Built on by (most cited first, 5 of 5)

- [Interpretable and Explainable Machine Learning Methods for Predictive Process Monitoring: A Systematic Literature Review](https://mlsanity.com/paper/2312.17584/) (2023)
- [XABPs: Towards eXplainable Autonomous Business Processes](https://mlsanity.com/paper/2507.23269/) (2025)
- [Is Machine Learning Unsafe and Irresponsible in Social Sciences? Paradoxes and Reconsidering from Recidivism Prediction Tasks](https://mlsanity.com/paper/2311.06537/) (2023)
- [Discussion: Effective and Interpretable Outcome Prediction by Training Sparse Mixtures of Linear Experts](https://mlsanity.com/paper/2407.13526/) (2024)
- [The Role of Explanation Styles and Perceived Accuracy on Decision Making in Predictive Process Monitoring](https://mlsanity.com/paper/2506.16617/) (2025)

## Abstract (arXiv)

Although a recent shift has been made in the field of predictive process monitoring to use models from the explainable artificial intelligence field, the evaluation still occurs mainly through performance-based metrics, thus not accounting for the actionability and implications of the explanations. In this paper, we define explainability through the interpretability of the explanations and the faithfulness of the explainability model in the field of process outcome prediction. The introduced properties are analysed along the event, case, and control flow perspective which are typical for a process-based analysis. This allows comparing inherently created explanations with post-hoc explanations. We benchmark seven classifiers on thirteen real-life events logs, and these cover a range of transparent and non-transparent machine learning and deep learning models, further complemented with explainability techniques. Next, this paper contributes a set of guidelines named X-MOP which allows selecting the appropriate model based on the event log specifications, by providing insight into how the varying preprocessing, model complexity and explainability techniques typical in process outcome prediction influence the explainability of the model.