# How well can a large language model explain business processes as perceived by users?

arXiv: https://arxiv.org/abs/2401.12846 · HTML: https://mlsanity.com/paper/2401.12846/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- [Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation](https://mlsanity.com/paper/2305.07375/) (2023)
- [Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations](https://mlsanity.com/paper/1912.09024/) (2019)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [AI-Augmented Business Process Management Systems: A Research Manifesto](https://mlsanity.com/paper/2201.12855/) (2022)
- [Explainable Predictive Process Monitoring](https://mlsanity.com/paper/2008.01807/) (2020)
- [Large Language Models for Business Process Management: Opportunities and Challenges](https://mlsanity.com/paper/2304.04309/) (2023)
- [Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Study](https://mlsanity.com/paper/2307.02194/) (2023)
- [Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction](https://mlsanity.com/paper/2105.07111/) (2021)
- [Process Mining Meets Causal Machine Learning: Discovering Causal Rules from Event Logs](https://mlsanity.com/paper/2009.01561/) (2020)
- [Multi-Dimensional Event Data in Graph Databases](https://mlsanity.com/paper/2005.14552/) (2020)
- [ProcessGPT: Transforming Business Process Management with Generative Artificial Intelligence](https://mlsanity.com/paper/2306.01771/) (2023)
- [An Explainable Decision Support System for Predictive Process Analytics](https://mlsanity.com/paper/2207.12782/) (2022)
- [A Case for Business Process-Specific Foundation Models](https://mlsanity.com/paper/2210.14739/) (2022)

## Built on by (most cited first, 8 of 8)

- [PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks](https://mlsanity.com/paper/2407.13244/) (2024)
- [Automating the Enterprise with Foundation Models](https://mlsanity.com/paper/2405.03710/) (2024)
- [WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks](https://mlsanity.com/paper/2406.13264/) (2024)
- [Agentic Business Process Management: A Research Manifesto](https://mlsanity.com/paper/2603.18916/) (2026)
- [Towards a Benchmark for Causal Business Process Reasoning with LLMs](https://mlsanity.com/paper/2406.05506/) (2024)
- [Selecting the Right LLM for eGov Explanations](https://mlsanity.com/paper/2504.21032/) (2025)
- [XABPs: Towards eXplainable Autonomous Business Processes](https://mlsanity.com/paper/2507.23269/) (2025)
- [Monetizing Currency Pair Sentiments through LLM Explainability](https://mlsanity.com/paper/2407.19922/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) are trained on a vast amount of text to interpret and generate human-like textual content. They are becoming a vital vehicle in realizing the vision of the autonomous enterprise, with organizations today actively adopting LLMs to automate many aspects of their operations. LLMs are likely to play a prominent role in future AI-augmented business process management systems, catering functionalities across all system lifecycle stages. One such system's functionality is Situation-Aware eXplainability (SAX), which relates to generating causally sound and human-interpretable explanations. In this paper, we present the SAX4BPM framework developed to generate SAX explanations. The SAX4BPM suite consists of a set of services and a central knowledge repository. The functionality of these services is to elicit the various knowledge ingredients that underlie SAX explanations. A key innovative component among these ingredients is the causal process execution view. In this work, we integrate the framework with an LLM to leverage its power to synthesize the various input ingredients for the sake of improved SAX explanations. Since the use of LLMs for SAX is also accompanied by a certain degree of doubt related to its capacity to adequately fulfill SAX along with its tendency for hallucination and lack of inherent capacity to reason, we pursued a methodological evaluation of the perceived quality of the generated explanations. We developed a designated scale and conducted a rigorous user study. Our findings show that the input presented to the LLMs aided with the guard-railing of its performance, yielding SAX explanations having better-perceived fidelity. This improvement is moderated by the perception of trust and curiosity. More so, this improvement comes at the cost of the perceived interpretability of the explanation.