# Leveraging Generative AI for Extracting Process Models from Multimodal Documents

arXiv: https://arxiv.org/abs/2406.04959 · HTML: https://mlsanity.com/paper/2406.04959/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2024.

## Built on by (most cited first, 4 of 4)

- [Conversational Process Model Redesign](https://mlsanity.com/paper/2505.05453/) (2025)
- [Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results](https://mlsanity.com/paper/2503.13520/) (2025)
- [What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models](https://mlsanity.com/paper/2507.11356/) (2025)
- [Automatic Generation of Executable BPMN Models from Medical Guidelines](https://mlsanity.com/paper/2604.07817/) (2026)

## Abstract (arXiv)

This paper presents an investigation of the capabilities of Generative Pre-trained Transformers (GPTs) to auto-generate graphical process models from multi-modal (i.e., text- and image-based) inputs. More precisely, we first introduce a small dataset as well as a set of evaluation metrics that allow for a ground truth-based evaluation of multi-modal process model generation capabilities. We then conduct an initial evaluation of commercial GPT capabilities using zero-, one-, and few-shot prompting strategies. Our results indicate that GPTs can be useful tools for semi-automated process modeling based on multi-modal inputs. More importantly, the dataset and evaluation metrics as well as the open-source evaluation code provide a structured framework for continued systematic evaluations moving forward.