# FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding

arXiv: https://arxiv.org/abs/2407.05183 · HTML: https://mlsanity.com/paper/2407.05183/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 6 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 10 of 10)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [VIS30K: A Collection of Figures and Tables from IEEE Visualization Conference Publications](https://mlsanity.com/paper/2101.01036/) (2020)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [Structure-aware Visualization Retrieval](https://mlsanity.com/paper/2202.05960/) (2022)
- [Recent, rapid advancement in visual question answering architecture: a review](https://mlsanity.com/paper/2203.01322/) (2022)

## Built on by (most cited first, 27 of 27)

- [Do Vision-Language Models Really Understand Visual Language?](https://mlsanity.com/paper/2410.00193/) (2024)
- [SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents](https://mlsanity.com/paper/2410.21155/) (2024)
- [Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models](https://mlsanity.com/paper/2503.11519/) (2025)
- [Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding](https://mlsanity.com/paper/2412.16420/) (2024)
- [VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing](https://mlsanity.com/paper/2605.15677/) (2026)
- [Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents](https://mlsanity.com/paper/2506.01344/) (2025)
- [Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding](https://mlsanity.com/paper/2505.07864/) (2025)
- [A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documents](https://mlsanity.com/paper/2507.22938/) (2025)
- [Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions](https://mlsanity.com/paper/2502.04389/) (2025)
- [DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition](https://mlsanity.com/paper/2504.04616/) (2025)
- [TechING: Towards Real World Technical Image Understanding via VLMs](https://mlsanity.com/paper/2601.18238/) (2026)
- [Guideline2Graph: Profile-Aware Multimodal Parsing for Executable Clinical Decision Graphs](https://mlsanity.com/paper/2604.02477/) (2026)
- [Procedural Knowledge Extraction from Industrial Troubleshooting Guides Using Vision Language Models](https://mlsanity.com/paper/2601.22754/) (2026)
- [Orchestration-Free Customer Service Automation: A Privacy-Preserving and Flowchart-Guided Framework](https://mlsanity.com/paper/2602.15377/) (2026)
- [Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models](https://mlsanity.com/paper/2603.02865/) (2026)
- [EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineering](https://mlsanity.com/paper/2605.27332/) (2026)
- [PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides](https://mlsanity.com/paper/2609.29718/) (2026)
- [JSynFlow: Japanese Synthesised Flowchart Visual Question Answering Dataset built with Large Language Models](https://mlsanity.com/paper/2602.04142/) (2026)
- [An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation](https://mlsanity.com/paper/2602.13376/) (2026)
- [FloCA: Towards Faithful and Logically Consistent Flowchart Reasoning](https://mlsanity.com/paper/2602.14035/) (2026)
- [DAGverse: Building Document-Grounded Semantic DAGs from Scientific Papers](https://mlsanity.com/paper/2603.25293/) (2026)
- [FlowExtract: Procedural Knowledge Extraction from Maintenance Flowcharts](https://mlsanity.com/paper/2604.06770/) (2026)
- [Prefix-Adaptive Block Diffusion for Efficient Document Recognition](https://mlsanity.com/paper/2605.16861/) (2026)
- [ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams](https://mlsanity.com/paper/2607.24707/) (2026)
- [A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images](https://mlsanity.com/paper/2608.14075/) (2026)
- [Back2Struct: Making Structured Images Editable Again](https://mlsanity.com/paper/2609.37016/) (2026)
- [Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models](https://mlsanity.com/paper/2610.04721/) (2026)

## Abstract (arXiv)

Flowcharts are graphical tools for representing complex concepts in concise visual representations. This paper introduces the FlowLearn dataset, a resource tailored to enhance the understanding of flowcharts. FlowLearn contains complex scientific flowcharts and simulated flowcharts. The scientific subset contains 3,858 flowcharts sourced from scientific literature and the simulated subset contains 10,000 flowcharts created using a customizable script. The dataset is enriched with annotations for visual components, OCR, Mermaid code representation, and VQA question-answer pairs. Despite the proven capabilities of Large Vision-Language Models (LVLMs) in various visual understanding tasks, their effectiveness in decoding flowcharts - a crucial element of scientific communication - has yet to be thoroughly investigated. The FlowLearn test set is crafted to assess the performance of LVLMs in flowchart comprehension. Our study thoroughly evaluates state-of-the-art LVLMs, identifying existing limitations and establishing a foundation for future enhancements in this relatively underexplored domain. For instance, in tasks involving simulated flowcharts, GPT-4V achieved the highest accuracy (58%) in counting the number of nodes, while Claude recorded the highest accuracy (83%) in OCR tasks. Notably, no single model excels in all tasks within the FlowLearn framework, highlighting significant opportunities for further development.