# Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding

arXiv: https://arxiv.org/abs/2505.07864 · HTML: https://mlsanity.com/paper/2505.07864/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 9 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 16 of 16)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs](https://mlsanity.com/paper/2406.18521/) (2024)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning](https://mlsanity.com/paper/2307.09474/) (2023)
- [ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering](https://mlsanity.com/paper/2405.07001/) (2024)
- [RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension](https://mlsanity.com/paper/2308.02299/) (2023)
- [FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding](https://mlsanity.com/paper/2407.05183/) (2024)
- [Towards Making Flowchart Images Machine Interpretable](https://mlsanity.com/paper/2501.17441/) (2025)
- [GRCNN: Graph Recognition Convolutional Neural Network for Synthesizing Programs from Flow Charts](https://mlsanity.com/paper/2011.05980/) (2020)

## Built on by (most cited first, 3 of 3)

- [Survey of GenAI for Automotive Software Development: From Requirements to Executable Code](https://mlsanity.com/paper/2507.15025/) (2025)
- [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026)
- [EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineering](https://mlsanity.com/paper/2605.27332/) (2026)

## Abstract (arXiv)

Flowcharts are indispensable tools in software design and business-process analysis, yet current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology that set these diagrams apart from natural images. We introduce a seven-stage pipeline grouped into three broader processes: (1) arrow-aware detection of nodes and arrow endpoints; (2) optical character recognition (OCR) to extract node text; and (3) construction of a structured prompt that guides the VLMs. Tested on a 90-question benchmark distilled from 30 annotated flowcharts, the method raises overall accuracy from 80 % to 89 % (+9 percentage points) without any task-specific fine-tuning. The gain is most pronounced for next-step queries (25/30 -> 30/30; 100 %, +17 pp); branch-result questions improve more modestly, and before-step questions remain difficult. A parallel evaluation with an LLM-as-a-Judge protocol shows the same trends, reinforcing the advantage of explicit arrow encoding. Limitations include dependence on detector and OCR precision, the small evaluation set, and residual errors at nodes with multiple incoming edges. Future work will enlarge the benchmark with synthetic and handwritten flowcharts and assess the approach on Business Process Model and Notation (BPMN) and Unified Modeling Language (UML).