# VizExtract: Automatic Relation Extraction from Data Visualizations

arXiv: https://arxiv.org/abs/2112.03485 · HTML: https://mlsanity.com/paper/2112.03485/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 22 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 4 of 4)

- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Data Extraction from Charts via Single Deep Neural Network](https://mlsanity.com/paper/1906.11906/) (2019)
- [Chart-Text: A Fully Automated Chart Image Descriptor](https://mlsanity.com/paper/1812.10636/) (2018)
- [Scatteract: Automated extraction of data from scatter plots](https://mlsanity.com/paper/1704.06687/) (2017)

## Built on by (most cited first, 2 of 2)

- [Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plots](https://mlsanity.com/paper/2503.12326/) (2025)
- [The Truth, the Whole Truth, and Nothing but the Truth: Automatic Visualization Evaluation from Reconstruction Quality](https://mlsanity.com/paper/2603.16873/) (2026)

## Abstract (arXiv)

Visual graphics, such as plots, charts, and figures, are widely used to communicate statistical conclusions. Extracting information directly from such visualizations is a key sub-problem for effective search through scientific corpora, fact-checking, and data extraction. This paper presents a framework for automatically extracting compared variables from statistical charts. Due to the diversity and variation of charting styles, libraries, and tools, we leverage a computer vision based framework to automatically identify and localize visualization facets in line graphs, scatter plots, or bar graphs and can include multiple series per graph. The framework is trained on a large synthetically generated corpus of matplotlib charts and we evaluate the trained model on other chart datasets. In controlled experiments, our framework is able to classify, with 87.5% accuracy, the correlation between variables for graphs with 1-3 series per graph, varying colors, and solid line styles. When deployed on real-world graphs scraped from the internet, it achieves 72.8% accuracy (81.2% accuracy when excluding "hard" graphs). When deployed on the FigureQA dataset, it achieves 84.7% accuracy.