# Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plots

arXiv: https://arxiv.org/abs/2503.12326 · HTML: https://mlsanity.com/paper/2503.12326/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 10 of 10)

- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Data Extraction from Charts via Single Deep Neural Network](https://mlsanity.com/paper/1906.11906/) (2019)
- [Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis](https://mlsanity.com/paper/2402.01730/) (2024)
- [From Text to Insight: Large Language Models for Materials Science Data Extraction](https://mlsanity.com/paper/2407.16867/) (2024)
- [Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis](https://mlsanity.com/paper/2405.00876/) (2024)
- [Image and Data Mining in Reticular Chemistry Using GPT-4V](https://mlsanity.com/paper/2312.05468/) (2023)
- [VizExtract: Automatic Relation Extraction from Data Visualizations](https://mlsanity.com/paper/2112.03485/) (2021)
- [Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis](https://mlsanity.com/paper/2501.15370/) (2025)

## Built on by (most cited first, 7 of 7)

- [LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature](https://mlsanity.com/paper/2510.26824/) (2025)
- [From Knowledge to Action: Outcomes of the 2025 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistry](https://mlsanity.com/paper/2605.03205/) (2026)
- [Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy](https://mlsanity.com/paper/2606.00065/) (2026)
- [TaskEval: Synthesised Evaluation for Foundation-Model Tasks](https://mlsanity.com/paper/2512.04442/) (2025)
- [Large language model-enabled automated data extraction for concrete materials informatics](https://mlsanity.com/paper/2604.22938/) (2026)
- [Material Database Agent: A Multimodal Agentic Framework for Scientific Literature Mining](https://mlsanity.com/paper/2605.04278/) (2026)
- [PlotPick: AI-powered batch extraction of numerical data from scientific figures](https://mlsanity.com/paper/2605.06021/) (2026)

## Abstract (arXiv)

Automated data extraction from research texts has been steadily improving, with the emergence of large language models (LLMs) accelerating progress even further. Extracting data from plots in research papers, however, has been such a complex task that it has predominantly been confined to manual data extraction. We show that current multimodal large language models, with proper instructions and engineered workflows, are capable of accurately extracting data from plots. This capability is inherent to the pretrained models and can be achieved with a chain-of-thought sequence of zero-shot engineered prompts we call PlotExtract, without the need to fine-tune. We demonstrate PlotExtract here and assess its performance on synthetic and published plots. We consider only plots with two axes in this analysis. For plots identified as extractable, PlotExtract finds points with over 90% precision (and around 90% recall) and errors in x and y position of around 5% or lower. These results prove that multimodal LLMs are a viable path for high-throughput data extraction for plots and in many circumstances can replace the current manual methods of data extraction.