# RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning

arXiv: https://arxiv.org/abs/2503.23131 · HTML: https://mlsanity.com/paper/2503.23131/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 37 of its 38 references that are in the map.
- First posted on arXiv 29 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 30 of 37)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [A Unified Sequence Interface for Vision Tasks](https://mlsanity.com/paper/2206.07669/) (2022)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models](https://mlsanity.com/paper/2404.13013/) (2024)
- [BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs](https://mlsanity.com/paper/2307.08581/) (2023)
- [LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models](https://mlsanity.com/paper/2312.02949/) (2023)
- [ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild](https://mlsanity.com/paper/2407.04172/) (2024)
- [A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding](https://mlsanity.com/paper/2407.01976/) (2024)
- [TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning](https://mlsanity.com/paper/2404.16635/) (2024)

## Built on by (most cited first, 11 of 11)

- [Towards Understanding Visual Grounding in Visual Language Models](https://mlsanity.com/paper/2509.10345/) (2025)
- [ChartM3: Benchmarking Chart Editing with Multimodal Instructions](https://mlsanity.com/paper/2507.21167/) (2025)
- [START: Spatial and Textual Learning for Chart Understanding](https://mlsanity.com/paper/2512.07186/) (2025)
- [MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment](https://mlsanity.com/paper/2508.17290/) (2025)
- [ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring](https://mlsanity.com/paper/2605.07415/) (2026)
- [ETCHR: Editing To Clarify and Harness Reasoning](https://mlsanity.com/paper/2605.23897/) (2026)
- [CycleChart: A Unified Consistency-Based Learning Framework for Bidirectional Chart Understanding and Generation](https://mlsanity.com/paper/2512.19173/) (2025)
- [VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought](https://mlsanity.com/paper/2603.11631/) (2026)
- [Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing](https://mlsanity.com/paper/2609.08657/) (2026)
- [DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis](https://mlsanity.com/paper/2609.32385/) (2026)
- [Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis](https://mlsanity.com/paper/2610.14173/) (2026)

## Abstract (arXiv)

Recently, Vision Language Models (VLMs) have increasingly emphasized document visual grounding to achieve better human-computer interaction, accessibility, and detailed understanding. However, its application to visualizations such as charts remains under-explored due to the inherent complexity of interleaved visual-numerical relationships in chart images. Existing chart understanding methods primarily focus on answering questions without explicitly identifying the visual elements that support their predictions. To bridge this gap, we introduce RefChartQA, a novel benchmark that integrates Chart Question Answering (ChartQA) with visual grounding, enabling models to refer elements at multiple granularities within chart images. Furthermore, we conduct a comprehensive evaluation by instruction-tuning 5 state-of-the-art VLMs across different categories. Our experiments demonstrate that incorporating spatial awareness via grounding improves response accuracy by over 15%, reducing hallucinations, and improving model reliability. Additionally, we identify key factors influencing text-spatial alignment, such as architectural improvements in TinyChart, which leverages a token-merging module for enhanced feature fusion. Our dataset is open-sourced for community development and further advancements. All models and code will be publicly available at https://github.com/moured/RefChartQA.