# Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks

arXiv: https://arxiv.org/abs/2407.02310 · HTML: https://mlsanity.com/paper/2407.02310/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Predicting Process Behaviour using Deep Learning](https://mlsanity.com/paper/1612.04600/) (2016)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [A systematic literature review on state-of-the-art deep learning methods for process prediction](https://mlsanity.com/paper/2101.09320/) (2021)
- [Process Modeling With Large Language Models](https://mlsanity.com/paper/2403.07541/) (2024)
- [Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Study](https://mlsanity.com/paper/2307.02194/) (2023)
- [SAP Signavio Academic Models: A Large Process Model Dataset](https://mlsanity.com/paper/2208.12223/) (2022)
- [Chit-Chat or Deep Talk: Prompt Engineering for Process Mining](https://mlsanity.com/paper/2307.09909/) (2023)
- [Multivariate Business Process Representation Learning utilizing Gramian Angular Fields and Convolutional Neural Networks](https://mlsanity.com/paper/2106.08027/) (2021)

## Built on by (most cited first, 14 of 14)

- [PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks](https://mlsanity.com/paper/2407.13244/) (2024)
- [Domain Adaptation of LLMs for Process Data](https://mlsanity.com/paper/2509.03161/) (2025)
- [Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding](https://mlsanity.com/paper/2504.00409/) (2025)
- [SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures](https://mlsanity.com/paper/2510.08942/) (2025)
- [On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks](https://mlsanity.com/paper/2504.21074/) (2025)
- [BPMN Assistant: An LLM-Based Approach to Business Process Modeling](https://mlsanity.com/paper/2509.24592/) (2025)
- [SAGE: A Service Agent Graph-guided Evaluation Benchmark](https://mlsanity.com/paper/2604.09285/) (2026)
- [Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation](https://mlsanity.com/paper/2505.22041/) (2025)
- [Evaluating LLM-Based Process Explanations under Progressive Behavioral-Input Reduction](https://mlsanity.com/paper/2510.09732/) (2025)
- [Exploring LLM Features in Predictive Process Monitoring for Small-Scale Event-Logs](https://mlsanity.com/paper/2601.11468/) (2026)
- [David vs. Goliath in Next Activity Prediction: Argmax vs. LSTM, Transformer, and LLM](https://mlsanity.com/paper/2606.15868/) (2026)
- [Revisiting Predictive Process Monitoring in the Age of Foundation Models: A Comparative Study of Sequence, Tabular, and LLM Approaches](https://mlsanity.com/paper/2607.27797/) (2026)
- [D-TAIA: Domain-Aware LLM Adaptation for Multi-Task Predictive Process Monitoring](https://mlsanity.com/paper/2608.28236/) (2026)
- [Automated Event Log Generation from Unstructured Text Using Finetuned LLMs](https://mlsanity.com/paper/2609.01320/) (2026)

## Abstract (arXiv)

The process mining community has recently recognized the potential of large language models (LLMs) for tackling various process mining tasks. Initial studies report the capability of LLMs to support process analysis and even, to some extent, that they are able to reason about how processes work. This latter property suggests that LLMs could also be used to tackle process mining tasks that benefit from an understanding of process behavior. Examples of such tasks include (semantic) anomaly detection and next activity prediction, which both involve considerations of the meaning of activities and their inter-relations. In this paper, we investigate the capabilities of LLMs to tackle such semantics-aware process mining tasks. Furthermore, whereas most works on the intersection of LLMs and process mining only focus on testing these models out of the box, we provide a more principled investigation of the utility of LLMs for process mining, including their ability to obtain process mining knowledge post-hoc by means of in-context learning and supervised fine-tuning. Concretely, we define three process mining tasks that benefit from an understanding of process semantics and provide extensive benchmarking datasets for each of them. Our evaluation experiments reveal that (1) LLMs fail to solve challenging process mining tasks out of the box and when provided only a handful of in-context examples, (2) but they yield strong performance when fine-tuned for these tasks, consistently surpassing smaller, encoder-based language models.