# Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature

arXiv: https://arxiv.org/abs/2412.12150 · HTML: https://mlsanity.com/paper/2412.12150/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2024.

## Built on by (most cited first, 9 of 9)

- [MMCR: Benchmarking Cross-Source Reasoning in Scientific Papers](https://mlsanity.com/paper/2503.16856/) (2025)
- [ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understanding](https://mlsanity.com/paper/2509.17481/) (2025)
- [Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents](https://mlsanity.com/paper/2506.01344/) (2025)
- [ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoning](https://mlsanity.com/paper/2512.14040/) (2025)
- [Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Data](https://mlsanity.com/paper/2503.16260/) (2025)
- [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026)
- [LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning](https://mlsanity.com/paper/2608.01328/) (2026)
- [Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks](https://mlsanity.com/paper/2510.06071/) (2025)
- [FloCA: Towards Faithful and Logically Consistent Flowchart Reasoning](https://mlsanity.com/paper/2602.14035/) (2026)

## Abstract (arXiv)

Scientific Literature charts often contain complex visual elements, including multi-plot figures, flowcharts, structural diagrams and etc. Evaluating multimodal models using these authentic and intricate charts provides a more accurate assessment of their understanding abilities. However, existing benchmarks face limitations: a narrow range of chart types, overly simplistic template-based questions and visual elements, and inadequate evaluation methods. These shortcomings lead to inflated performance scores that fail to hold up when models encounter real-world scientific charts. To address these challenges, we introduce a new benchmark, Scientific Chart QA (SCI-CQA), which emphasizes flowcharts as a critical yet often overlooked category. To overcome the limitations of chart variety and simplistic visual elements, we curated a dataset of 202,760 image-text pairs from 15 top-tier computer science conferences papers over the past decade. After rigorous filtering, we refined this to 37,607 high-quality charts with contextual information. SCI-CQA also introduces a novel evaluation framework inspired by human exams, encompassing 5,629 carefully curated questions, both objective and open-ended. Additionally, we propose an efficient annotation pipeline that significantly reduces data annotation costs. Finally, we explore context-based chart understanding, highlighting the crucial role of contextual information in solving previously unanswerable questions.