# Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results

arXiv: https://arxiv.org/abs/2503.13520 · HTML: https://mlsanity.com/paper/2503.13520/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [TrustLLM: Trustworthiness in Large Language Models](https://mlsanity.com/paper/2401.05561/) (2024)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2503.09572/) (2025)
- [xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://mlsanity.com/paper/2409.03215/) (2024)
- [A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages](https://mlsanity.com/paper/2410.03981/) (2024)
- [Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators](https://mlsanity.com/paper/2307.05532/) (2023)
- [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022)
- [Thought of Search: Planning with Language Models Through The Lens of Efficiency](https://mlsanity.com/paper/2404.11833/) (2024)
- [Process Modeling With Large Language Models](https://mlsanity.com/paper/2403.07541/) (2024)
- [Large Language Models for Business Process Management: Opportunities and Challenges](https://mlsanity.com/paper/2304.04309/) (2023)
- [Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning](https://mlsanity.com/paper/2311.09830/) (2023)
- [Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions](https://mlsanity.com/paper/2404.09135/) (2024)
- [Large Action Models: From Inception to Implementation](https://mlsanity.com/paper/2412.10047/) (2024)
- [PET: An Annotated Dataset for Process Extraction from Natural Language Text](https://mlsanity.com/paper/2203.04860/) (2022)
- [Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis](https://mlsanity.com/paper/2412.00023/) (2024)
- [SAP Signavio Academic Models: A Large Process Model Dataset](https://mlsanity.com/paper/2208.12223/) (2022)
- [Leveraging Large Language Models for Enhanced Process Model Comprehension](https://mlsanity.com/paper/2408.08892/) (2024)
- [Leveraging Generative AI for Extracting Process Models from Multimodal Documents](https://mlsanity.com/paper/2406.04959/) (2024)
- [Bridging Domain Knowledge and Process Discovery Using Large Language Models](https://mlsanity.com/paper/2408.17316/) (2024)
- [Evaluating Large Language Models for Material Selection](https://mlsanity.com/paper/2405.03695/) (2024)

## Built on by (most cited first, 1 of 1)

- [What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models](https://mlsanity.com/paper/2507.11356/) (2025)

## Abstract (arXiv)

Large language models (LLM) have revolutionized the processing of natural language. Although first benchmarks of the process modeling abilities of LLM are promising, it is currently under debate to what extent an LLM can generate good process models. In this contribution, we argue that the evaluation of the process modeling abilities of LLM is far from being trivial. Hence, available evaluation results must be taken carefully. For example, even in a simple scenario, not only the quality of a model should be taken into account, but also the costs and time needed for generation. Thus, an LLM does not generate one optimal solution, but a set of Pareto-optimal variants. Moreover, there are several further challenges which have to be taken into account, e.g. conceptualization of quality, validation of results, generalizability, and data leakage. We discuss these challenges in detail and discuss future experiments to tackle these challenges scientifically.