mlsanityData to 7 Oct 2026

First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Models

arXiv 2406.10057 ↗First posted 14 Jun 2024cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

With the development of Multimodal Large Language Models (MLLMs) technology, its general capabilities are increasingly powerful. To evaluate the various abilities of MLLMs, numerous evaluation systems have emerged. But now there is still a lack of a comprehensive method to evaluate MLLMs in the tasks related to flowcharts, which are very important in daily life and work. We propose the first comprehensive method, FlowCE, to assess MLLMs across various dimensions for tasks related to flowcharts. It encompasses evaluating MLLMs' abilities in Reasoning, Localization Recognition, Information Extraction, Logical Verification, and Summarization on flowcharts. However, we find that even the GPT4o model achieves only a score of 56.63. Among open-source models, Phi-3-Vision obtained the highest score of 49.97. We hope that FlowCE can contribute to future research on MLLMs for tasks based on flowcharts. urlhttps://github.com/360AILABNLP/FlowCE

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 44 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

37 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023A Survey on Multimodal Large Language Models23 Jun
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  8. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
16 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2024Yi: Open Foundation Models by 01.AIcited by 706
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2021InfographicVQAcited by 430
  9. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  10. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  11. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  12. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  13. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  14. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  15. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  16. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  2. 2026FloCA: Towards Faithful and Logically Consistent Flowchart Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding
  2. 2024A Survey on Benchmarks of Multimodal Large Language Models
  3. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  6. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems
  7. 2026LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
  8. 2024FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts