mlsanityData to 14 Oct 2026

Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding

arXiv 2505.07864 ↗First posted 9 May 2025cs.AI cs.CL cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Flowcharts are indispensable tools in software design and business-process analysis, yet current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology that set these diagrams apart from natural images. We introduce a seven-stage pipeline grouped into three broader processes: (1) arrow-aware detection of nodes and arrow endpoints; (2) optical character recognition (OCR) to extract node text; and (3) construction of a structured prompt that guides the VLMs. Tested on a 90-question benchmark distilled from 30 annotated flowcharts, the method raises overall accuracy from 80 % to 89 % (+9 percentage points) without any task-specific fine-tuning. The gain is most pronounced for next-step queries (25/30 -> 30/30; 100 %, +17 pp); branch-result questions improve more modestly, and before-step questions remain difficult. A parallel evaluation with an LLM-as-a-Judge protocol shows the same trends, reinforcing the advantage of explicit arrow encoding. Limitations include dependence on detector and OCR precision, the small evaluation set, and residual errors at nodes with multiple incoming edges. Future work will enlarge the benchmark with synthetic and handwritten flowcharts and assess the approach on Business Process Model and Notation (BPMN) and Unified Modeling Language (UML).

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 21 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

16 of 21 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding6 Jul
  2. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs26 Jun
  3. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering11 May
  4. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning19 Feb

Other references · 12

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  8. 2016A Diagram Is Worth A Dozen Imagescited by 804
  9. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  10. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehensioncited by 27
  11. 2025Towards Making Flowchart Images Machine Interpretablecited by 5
  12. 2020GRCNN: Graph Recognition Convolutional Neural Network for Synthesizing Programs from Flow Chartscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Survey of GenAI for Automotive Software Development: From Requirements to Executable Codecited by 5
  2. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  3. 2026EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineeringcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding
  2. 2026FlowExtract: Procedural Knowledge Extraction from Maintenance Flowcharts
  3. 2025Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents
  4. 2025Flowchart2Mermaid: A Vision-Language Model Powered System for Converting Flowcharts into Editable Diagram Code
  5. 2025Visual Programmability: A Guide for Code-as-Thought in Chart Understanding
  6. 2025Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions
  7. 2026An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation
  8. 2025Structured Extraction from Business Process Diagrams Using Vision-Language Models