mlsanityData to 14 Oct 2026

Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy

arXiv 2606.00065 ↗First posted 19 May 2026cs.IR cond-mat.mtrl-sci cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Automated extraction of materials composition-property data from scientific literature has advanced considerably with the development of large language model-based pipelines; however, existing frameworks remain limited to textual and tabular content, overlooking the substantial proportion of quantitative property data reported exclusively in scientific figures. Here, we extend ComProScanner, a fully end-to-end multi-agent framework for automated composition-property database construction, with a native vision-language model (VLM) based figure extraction capability. The extension introduces a FigureExtractor utility for caption-keyword-based figure filtering across all supported publishers, and a GraphExtractorTool agent that passes extracted figures to a configurable VLM to recover composition-property pairs from scientific charts and plots. Four VLMs are selected for evaluation on the basis of the LMArena Diagram leaderboard with an input cost criterion of less than 1.50 per million tokens. Benchmarking on 50 piezoelectric ceramic articles from the established d33 test corpus demonstrates that Gemini-3-Flash-Preview achieves the highest performance with a composition accuracy of 0.97 and a normalised F1 score of 0.97, whilst remaining the most cost-effective model among the four evaluated. We additionally introduce a range-based value error threshold parameter into the evaluation framework, providing a more physically meaningful assessment of numeric property values extracted from figures than exact value matching. These contributions establish VLM-integrated ComProScanner as the first materials-specific, fully automated, multimodal literature mining platform capable of extracting structured composition-property data from text, tables, and figures within a single unified pipeline.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 32 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

11 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Leveraging Vision Capabilities of Multimodal LLMs for Automated Data Extraction from Plots16 Mar
  2. 2024Probing the limitations of multimodal language models for chemistry and materials research25 Nov
  3. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec

Other references · 8

  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2025OpenAI GPT-5 System Cardcited by 989
  3. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  4. 2023Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineeringcited by 86
  5. 2023Agent-based Learning of Materials Datasets from Scientific Literaturecited by 15
  6. 2021Plot2Spectra: an Automatic Spectra Extraction Toolcited by 5
  7. 2023MatGD: Materials Graph Digitizercited by 4
  8. 2025ComProScanner: A multi-agent based framework for composition-property structured data extraction from scientific literaturecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SALSA: Semi-Autonomous Literature Summarization Assistantcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding
  2. 2024MatViX: Multimodal Information Extraction from Visually Rich Articles
  3. 2025Large language models in materials science and the need for open-source approaches
  4. 2024Mining experimental data from Materials Science literature with Large Language Models: an evaluation study
  5. 2025Automated Extraction of Material Properties using LLM-based AI Agents
  6. 2025MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis
  7. 2025Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models
  8. 2026MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models