mlsanityData to 7 Oct 2026

FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding

arXiv 2407.05183 ↗First posted 6 Jul 2024cs.CV cs.AIcited by 26

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Flowcharts are graphical tools for representing complex concepts in concise visual representations. This paper introduces the FlowLearn dataset, a resource tailored to enhance the understanding of flowcharts. FlowLearn contains complex scientific flowcharts and simulated flowcharts. The scientific subset contains 3,858 flowcharts sourced from scientific literature and the simulated subset contains 10,000 flowcharts created using a customizable script. The dataset is enriched with annotations for visual components, OCR, Mermaid code representation, and VQA question-answer pairs. Despite the proven capabilities of Large Vision-Language Models (LVLMs) in various visual understanding tasks, their effectiveness in decoding flowcharts - a crucial element of scientific communication - has yet to be thoroughly investigated. The FlowLearn test set is crafted to assess the performance of LVLMs in flowchart comprehension. Our study thoroughly evaluates state-of-the-art LVLMs, identifying existing limitations and establishing a foundation for future enhancements in this relatively underexplored domain. For instance, in tasks involving simulated flowcharts, GPT-4V achieved the highest accuracy (58%) in counting the number of nodes, while Claude recorded the highest accuracy (83%) in OCR tasks. Notably, no single model excels in all tasks within the FlowLearn framework, highlighting significant opportunities for further development.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 31 references that are in the map. First posted on arXiv 6 Jul 2024.

Builds on

10 of 31 references.

References · 10

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  4. 2020Towards Accurate Scene Text Recognition with Semantic Reasoning Networkscited by 131
  5. 2016Image-to-Markup Generation with Coarse-to-Fine Attentioncited by 79
  6. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  7. 2020VIS30K: A Collection of Figures and Tables from IEEE Visualization Conference Publicationscited by 13
  8. 2021Neural Attention for Image Captioning: Review of Outstanding Methodscited by 11
  9. 2022Structure-aware Visualization Retrievalcited by 11
  10. 2022Recent, rapid advancement in visual question answering architecture: a reviewcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 17.

26 papers.

Sort:

By area:

  1. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  2. 2024SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documentscited by 20
  3. 2025Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Modelscited by 12
  4. 2024Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understandingcited by 10
  5. 2026VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editingcited by 6
  6. 2025Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agentscited by 4
  7. 2025Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encodingcited by 3
  8. 2025A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documentscited by 3
  9. 2025DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognitioncited by 2
  10. 2026TechING: Towards Real World Technical Image Understanding via VLMscited by 2
  11. 2025Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutionscited by 1
  12. 2026Procedural Knowledge Extraction from Industrial Troubleshooting Guides Using Vision Language Modelscited by 1
14 more
  1. 2026Orchestration-Free Customer Service Automation: A Privacy-Preserving and Flowchart-Guided Frameworkcited by 1
  2. 2026Guideline2Graph: Profile-Aware Multimodal Parsing for Executable Clinical Decision Graphscited by 1
  3. 2026JSynFlow: Japanese Synthesised Flowchart Visual Question Answering Dataset built with Large Language Modelscited by 0
  4. 2026An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generationcited by 0
  5. 2026FloCA: Towards Faithful and Logically Consistent Flowchart Reasoningcited by 0
  6. 2026Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Modelscited by 0
  7. 2026DAGverse: Building Document-Grounded Semantic DAGs from Scientific Paperscited by 0
  8. 2026FlowExtract: Procedural Knowledge Extraction from Maintenance Flowchartscited by 0
  9. 2026Prefix-Adaptive Block Diffusion for Efficient Document Recognitioncited by 0
  10. 2026EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineeringcited by 0
  11. 2026ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagramscited by 0
  12. 2026A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Imagescited by 0
  13. 2026PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slidescited by 0
  14. 2026Back2Struct: Making Structured Images Editable Againcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Models
  2. 2024FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts
  3. 2025Improving Large Vision-Language Models' Understanding for Flow Field Data
  4. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs
  5. 2025On the Perception Bottleneck of VLMs for Chart Understanding
  6. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts
  7. 2025Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations
  8. 2025Flowchart2Mermaid: A Vision-Language Model Powered System for Converting Flowcharts into Editable Diagram Code