# Assessing the Business Process Modeling Competences of Large Language Models

arXiv: https://arxiv.org/abs/2601.21787 · HTML: https://mlsanity.com/paper/2601.21787/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks](https://mlsanity.com/paper/2407.13244/) (2024)
- [ProMoAI: Process Modeling with Generative AI](https://mlsanity.com/paper/2403.04327/) (2024)
- [WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks](https://mlsanity.com/paper/2406.13264/) (2024)
- [Towards a Benchmark for Causal Business Process Reasoning with LLMs](https://mlsanity.com/paper/2406.05506/) (2024)
- [LLM4PM: A case study on using Large Language Models for Process Modeling in Enterprise Organizations](https://mlsanity.com/paper/2407.17478/) (2024)
- [Bridging Domain Knowledge and Process Discovery Using Large Language Models](https://mlsanity.com/paper/2408.17316/) (2024)

## Built on by (most cited first, 1 of 1)

- [Large Language Models to Enhance Business Process Modeling: Past, Present, and Future Trends](https://mlsanity.com/paper/2604.14034/) (2026)

## Abstract (arXiv)

The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions. Recent advances in large language models (LLMs) have significantly expanded the possibilities for generating BPMN models directly from natural language, building upon earlier text-to-process methods with enhanced capabilities in handling complex descriptions. However, there is a lack of systematic evaluations of LLM-generated process models. Current efforts either use LLM-as-a-judge approaches or do not consider established dimensions of model quality. To this end, we introduce BEF4LLM, a novel LLM evaluation framework comprising four perspectives: syntactic quality, pragmatic quality, semantic quality, and validity. Using BEF4LLM, we conduct a comprehensive analysis of open-source LLMs and benchmark their performance against human modeling experts. Results indicate that LLMs excel in syntactic and pragmatic quality, while humans outperform LLMs in semantic aspects; however, the differences in scores are relatively modest, highlighting LLMs' competitive potential despite challenges in validity and semantic quality. The insights highlight current strengths and limitations of using LLMs for BPMN modeling and guide future model development and fine-tuning. Addressing these areas is essential for advancing the practical deployment of LLMs in business process modeling.