# Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis

arXiv: https://arxiv.org/abs/2412.00023 · HTML: https://mlsanity.com/paper/2412.00023/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2024.

## Built on by (most cited first, 13 of 13)

- [SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures](https://mlsanity.com/paper/2510.08942/) (2025)
- [On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks](https://mlsanity.com/paper/2504.21074/) (2025)
- [BPMN Assistant: An LLM-Based Approach to Business Process Modeling](https://mlsanity.com/paper/2509.24592/) (2025)
- [Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results](https://mlsanity.com/paper/2503.13520/) (2025)
- [What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models](https://mlsanity.com/paper/2507.11356/) (2025)
- [Knowledge-Driven Hallucination in Large Language Models: An Empirical Study on Process Modeling](https://mlsanity.com/paper/2509.15336/) (2025)
- [Specification-Driven Generation and Evaluation of Discrete-Event World Models via the DEVS Formalism](https://mlsanity.com/paper/2603.03784/) (2026)
- [Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts](https://mlsanity.com/paper/2603.12895/) (2026)
- [Evaluation of LLMs for Process Model Analysis and Optimization](https://mlsanity.com/paper/2510.07489/) (2025)
- [Instruction-Tuning Open-Weight Language Models for BPMN Model Generation](https://mlsanity.com/paper/2512.12063/) (2025)
- [Large Language Models to Enhance Business Process Modeling: Past, Present, and Future Trends](https://mlsanity.com/paper/2604.14034/) (2026)
- [Beyond Control-Flow: Integrating the Resource Perspective into Multi-Collaborative Process Modeling from Text](https://mlsanity.com/paper/2605.24546/) (2026)
- [A Hybrid LLM-Based Framework for Automated Security Annotation Generation in Business Process Models](https://mlsanity.com/paper/2608.14370/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are rapidly transforming various fields, and their potential in Business Process Management (BPM) is substantial. This paper assesses the capabilities of LLMs on business process modeling using a framework for automating this task, a comprehensive benchmark, and an analysis of LLM self-improvement strategies. We present a comprehensive evaluation of 16 state-of-the-art LLMs from major AI vendors using a custom-designed benchmark of 20 diverse business processes. Our analysis highlights significant performance variations across LLMs and reveals a positive correlation between efficient error handling and the quality of generated models. It also shows consistent performance trends within similar LLM groups. Furthermore, we investigate LLM self-improvement techniques, encompassing self-evaluation, input optimization, and output optimization. Our findings indicate that output optimization, in particular, offers promising potential for enhancing quality, especially in models with initially lower performance. Our contributions provide insights for leveraging LLMs in BPM, paving the way for more advanced and automated process modeling techniques.