mlsanityData to 14 Oct 2026

MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature

arXiv 2606.29667 ↗First posted 29 Jun 2026cs.CV cond-mat.mtrl-sci cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

29 of 49 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2018YOLOv3: An Incremental Improvementcited by 2,659
  9. 2017Billion-scale similarity search with GPUscited by 2,497
  10. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  11. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 515
  12. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 425
17 more
  1. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 305
  2. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  3. 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 177
  4. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  5. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literaturecited by 34
  6. 2023GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captionscited by 21
  7. 2017A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networkscited by 10
  8. 2025Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learningcited by 9
  9. 2025Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterizationcited by 9
  10. 2024MatViX: Multimodal Information Extraction from Visually Rich Articlescited by 8
  11. 2026Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectoriecited by 5
  12. 2026OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understandingcited by 5
  13. 2016Automatic Separation of Compound Figures in Scientific Articlescited by 4
  14. 2025Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023cited by 4
  15. 2025MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysiscited by 3
  16. 2026MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Sciencecited by 3
  17. 2025Leveraging Author-Specific Context for Scientific Figure Caption Generation: 3rd SciCap Challengecited by 1

Built on by

Not cited yet.