# Evaluation of LLMs for Process Model Analysis and Optimization

arXiv: https://arxiv.org/abs/2510.07489 · HTML: https://mlsanity.com/paper/2510.07489/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [Large Language Models for Business Process Management: Opportunities and Challenges](https://mlsanity.com/paper/2304.04309/) (2023)
- [Just Tell Me: Prompt Engineering in Business Process Management](https://mlsanity.com/paper/2304.07183/) (2023)
- [Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis](https://mlsanity.com/paper/2412.00023/) (2024)

## Abstract (arXiv)

In this paper, we report our experience with several LLMs for their ability to understand a process model in an interactive, conversational style, find syntactical and logical errors in it, and reason with it in depth through a natural language (NL) interface. Our findings show that a vanilla, untrained LLM like ChatGPT (model o3) in a zero-shot setting is effective in understanding BPMN process models from images and answering queries about them intelligently at syntactic, logic, and semantic levels of depth. Further, different LLMs vary in performance in terms of their accuracy and effectiveness. Nevertheless, our empirical analysis shows that LLMs can play a valuable role as assistants for business process designers and users. We also study the LLM's "thought process" and ability to perform deeper reasoning in the context of process analysis and optimization. We find that the LLMs seem to exhibit anthropomorphic properties.