# SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding

arXiv: https://arxiv.org/abs/2506.21319 · HTML: https://mlsanity.com/paper/2506.21319/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 17 of 17)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning](https://mlsanity.com/paper/2404.16635/) (2024)
- [VizML: A Machine Learning Approach to Visualization Recommendation](https://mlsanity.com/paper/1808.04819/) (2018)
- [VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository](https://mlsanity.com/paper/1905.04616/) (2019)
- [FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System](https://mlsanity.com/paper/1908.00681/) (2019)
- [Iris: A Conversational Agent for Complex Tasks](https://mlsanity.com/paper/1707.05015/) (2017)
- [Beagle: Automated Extraction and Interpretation of Visualizations from the Web](https://mlsanity.com/paper/1711.05962/) (2017)
- [Searching the Visual Style and Structure of D3 Visualizations](https://mlsanity.com/paper/1907.11265/) (2019)
- [mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning](https://mlsanity.com/paper/2404.01548/) (2024)
- [VisImages: A Fine-Grained Expert-Annotated Visualization Dataset](https://mlsanity.com/paper/2007.04584/) (2020)
- [OldVisOnline: Curating a Dataset of Historical Visualizations](https://mlsanity.com/paper/2308.16053/) (2023)
- [ZuantuSet: A Collection of Historical Chinese Visualizations and Illustrations](https://mlsanity.com/paper/2502.19093/) (2025)

## Built on by (most cited first, 5 of 5)

- [Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics](https://mlsanity.com/paper/2603.05832/) (2026)
- [Generating Statistical Charts with Validation-Driven LLM Workflows](https://mlsanity.com/paper/2605.00800/) (2026)
- [From Static to Interactive: Authoring Interactive Visualizations via Natural Language](https://mlsanity.com/paper/2601.17736/) (2026)
- [EvidenceLens: A Claim-Evidence Matrix for Auditing Financial Question Answering](https://mlsanity.com/paper/2606.23724/) (2026)
- [Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents](https://mlsanity.com/paper/2609.17842/) (2026)

## Abstract (arXiv)

Current multimodal large language models (MLLMs), while effective in natural image understanding, struggle with visualization understanding due to their inability to decode the data-to-visual mapping and extract structured information. To address these challenges, we propose SimVec, a novel simplified vector format that encodes chart elements such as mark type, position, and size. The effectiveness of SimVec is demonstrated by using MLLMs to reconstruct chart information from SimVec formats. Then, we build a new visualization dataset, SimVecVis, to enhance the performance of MLLMs in visualization understanding, which consists of three key dimensions: bitmap images of charts, their SimVec representations, and corresponding data-centric question-answering (QA) pairs with explanatory chain-of-thought (CoT) descriptions. We finetune state-of-the-art MLLMs (e.g., MiniCPM and Qwen-VL), using SimVecVis with different dataset dimensions. The experimental results show that it leads to substantial performance improvements of MLLMs with good spatial perception capabilities (e.g., MiniCPM) in data-centric QA tasks. Our dataset and source code are available at: https://github.com/VIDA-Lab/SimVecVis.