# Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy

arXiv: https://arxiv.org/abs/2606.00065 · HTML: https://mlsanity.com/paper/2606.00065/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 19 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 11 of 11)

- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [Probing the limitations of multimodal language models for chemistry and materials research](https://mlsanity.com/paper/2411.16955/) (2024)
- [Agent-based Learning of Materials Datasets from Scientific Literature](https://mlsanity.com/paper/2312.11690/) (2023)
- [Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plots](https://mlsanity.com/paper/2503.12326/) (2025)
- [Plot2Spectra: an Automatic Spectra Extraction Tool](https://mlsanity.com/paper/2107.02827/) (2021)
- [MatGD: Materials Graph Digitizer](https://mlsanity.com/paper/2311.12806/) (2023)
- [ComProScanner: A multi-agent based framework for composition-property structured data extraction from scientific literature](https://mlsanity.com/paper/2510.20362/) (2025)

## Built on by (most cited first, 1 of 1)

- [SALSA: Semi-Autonomous Literature Summarization Assistant](https://mlsanity.com/paper/2609.22210/) (2026)

## Abstract (arXiv)

Automated extraction of materials composition-property data from scientific literature has advanced considerably with the development of large language model-based pipelines; however, existing frameworks remain limited to textual and tabular content, overlooking the substantial proportion of quantitative property data reported exclusively in scientific figures. Here, we extend ComProScanner, a fully end-to-end multi-agent framework for automated composition-property database construction, with a native vision-language model (VLM) based figure extraction capability. The extension introduces a FigureExtractor utility for caption-keyword-based figure filtering across all supported publishers, and a GraphExtractorTool agent that passes extracted figures to a configurable VLM to recover composition-property pairs from scientific charts and plots. Four VLMs are selected for evaluation on the basis of the LMArena Diagram leaderboard with an input cost criterion of less than 1.50 per million tokens. Benchmarking on 50 piezoelectric ceramic articles from the established d33 test corpus demonstrates that Gemini-3-Flash-Preview achieves the highest performance with a composition accuracy of 0.97 and a normalised F1 score of 0.97, whilst remaining the most cost-effective model among the four evaluated. We additionally introduce a range-based value error threshold parameter into the evaluation framework, providing a more physically meaningful assessment of numeric property values extracted from figures than exact value matching. These contributions establish VLM-integrated ComProScanner as the first materials-specific, fully automated, multimodal literature mining platform capable of extracting structured composition-property data from text, tables, and figures within a single unified pipeline.