mlsanityData to 14 Oct 2026

PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks

arXiv 2407.13244 ↗First posted 18 Jul 2024cs.CL cs.DBcited by 16

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have the potential to semi-automate some process mining (PM) analyses. While commercial models are already adequate for many analytics tasks, the competitive level of open-source LLMs in PM tasks is unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive benchmark for PM focusing on domain knowledge (process-mining-specific and process-specific) and on different implementation strategies. We focus also on the challenges in creating such a benchmark, related to the public availability of the data and on evaluation biases by the LLMs. Overall, we observe that most of the considered LLMs can perform some process mining tasks at a satisfactory level, but tiny models that would run on edge devices are still inadequate. We also conclude that while the proposed benchmark is useful for identifying LLMs that are adequate for process mining tasks, further research is needed to overcome the evaluation biases and perform a more thorough ranking of the competitive LLMs.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

19 of 30 references.

References · 19

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  3. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  4. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  5. 2024Can LLM be a Personalized Judge?cited by 100
  6. 2023Large Language Models can accomplish Business Process Management Taskscited by 34
  7. 2024Process Modeling With Large Language Modelscited by 26
  8. 2023Large Language Models for Business Process Management: Opportunities and Challengescited by 24
  9. 2023Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Studycited by 20
  10. 2024Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Taskscited by 14
  11. 2024ProMoAI: Process Modeling with Generative AIcited by 12
  12. 2024WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Taskscited by 11
7 more
  1. 2023Chit-Chat or Deep Talk: Prompt Engineering for Process Miningcited by 9
  2. 2023Leveraging Large Language Models (LLMs) for Process Mining (Technical Report)cited by 9
  3. 2024How well can a large language model explain business processes as perceived by users?cited by 8
  4. 2024Ranking Large Language Models without Ground Truthcited by 8
  5. 2024Towards a Benchmark for Causal Business Process Reasoning with LLMscited by 7
  6. 2023A Collection of Simulated Event Logs for Fairness Assessment in Process Miningcited by 4
  7. 2024Evaluating Large Language Models in Process Mining: Capabilities, Benchmarks, and Evaluation Strategiescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 7.

16 papers.

Sort:

By area:

  1. 2024PersoBench: Benchmarking Personalized Response Generation in Large Language Modelscited by 13
  2. 2025Domain Adaptation of LLMs for Process Datacited by 6
  3. 2025SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedurescited by 4
  4. 2025Automated Business Process Analysis: An LLM-Based Approach to Value Assessmentcited by 2
  5. 2025On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Taskscited by 2
  6. 2025LLMs that Understand Processes: Instruction-tuning for Semantics-Aware Process Miningcited by 2
  7. 2026SAGE: A Service Agent Graph-guided Evaluation Benchmarkcited by 2
  8. 2025On LLM-Assisted Generation of Smart Contracts from Business Processescited by 1
  9. 2026Assessing the Business Process Modeling Competences of Large Language Modelscited by 1
  10. 2025No AI Without PI! Object-Centric Process Mining as the Enabler for Generative, Predictive, and Prescriptive Artificial Intelligencecited by 0
  11. 2025Evaluating LLM-Based Process Explanations under Progressive Behavioral-Input Reductioncited by 0
  12. 2026Exploring LLM Features in Predictive Process Monitoring for Small-Scale Event-Logscited by 0
4 more
  1. 2026Beyond Generalist LLMs: Specialist Agentic Systems for Structured Code Workflow Executioncited by 0
  2. 2026Revisiting Predictive Process Monitoring in the Age of Foundation Models: A Comparative Study of Sequence, Tabular, and LLM Approachescited by 0
  3. 2026PetriBench: Benchmarking LLM Reasoning over Dynamic State Spacescited by 0
  4. 2026Goal-driven Variant Categorizationcited by 0