# Structured Extraction from Business Process Diagrams Using Vision-Language Models

arXiv: https://arxiv.org/abs/2511.22448 · HTML: https://mlsanity.com/paper/2511.22448/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 51 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023)
- [ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language](https://mlsanity.com/paper/2311.01920/) (2023)
- [ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2403.09028/) (2024)
- [Large Language Models can accomplish Business Process Management Tasks](https://mlsanity.com/paper/2307.09923/) (2023)
- [FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts](https://mlsanity.com/paper/2406.19237/) (2024)
- [Image2Struct: Benchmarking Structure Extraction for Vision-Language Models](https://mlsanity.com/paper/2410.22456/) (2024)
- [Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction](https://mlsanity.com/paper/2411.03707/) (2024)
- [ProMoAI: Process Modeling with Generative AI](https://mlsanity.com/paper/2403.04327/) (2024)
- [Opus: A Large Work Model for Complex Workflow Generation](https://mlsanity.com/paper/2412.00573/) (2024)

## Built on by (most cited first, 3 of 3)

- [EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineering](https://mlsanity.com/paper/2605.27332/) (2026)
- [Unlocking UML Class Diagram Understanding in Vision Language Models](https://mlsanity.com/paper/2605.11634/) (2026)
- [LLM-based Vulnerability Discovery in Business Process Documentation](https://mlsanity.com/paper/2608.04271/) (2026)

## Abstract (arXiv)

Business Process Model and Notation (BPMN) is a widely adopted standard for representing complex business workflows. While BPMN diagrams are often exchanged as visual images, existing methods primarily rely on XML representations for computational analysis. In this work, we present a pipeline that leverages Vision-Language Models (VLMs) to extract structured JSON representations of BPMN diagrams directly from images, without requiring source model files or textual annotations. We also incorporate optical character recognition (OCR) for textual enrichment and evaluate the generated element lists against ground truth data derived from the source XML files. Our approach enables robust component extraction in scenarios where original source files are unavailable. We benchmark multiple VLMs and observe performance improvements in several models when OCR is used for text enrichment. In addition, we conducted extensive statistical analyses of OCR-based enrichment methods and prompt ablation studies, providing a clearer understanding of their impact on model performance.