mlsanityData to 7 Oct 2026

S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understanding

arXiv 2601.00264 ↗First posted 1 Jan 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Multimodal learning has revolutionized general domain tasks, yet its application in scientific discovery is hindered by the profound semantic gap between complex scientific imagery and sparse textual descriptions. We present S1-MMAlign, a large-scale, multi-disciplinary multimodal dataset comprising over 15.5 million high-quality image-text pairs derived from 2.5 million open-access scientific papers. Spanning disciplines from physics and biology to engineering, the dataset captures diverse visual modalities including experimental setups, heatmaps, and microscopic imagery. To address the pervasive issue of weak alignment in raw scientific captions, we introduce an AI-ready semantic enhancement pipeline that leverages advanced multimodal large language models to recaption images, by synthesizing comprehensive context from paper abstracts and the citation contexts of corresponding figures. Technical validation confirms that our enhancement pipeline markedly improves data quality via reduced SciBERT pseudo-perplexity and enhanced CLIP image-text alignment, while also significantly boosting multimodal large language models performance in zero-shot scientific captioning, multi-domain scientific reasoning, and visual instruction tuning. S1-MMAlign provides a pivotal foundational resource for cross-modal scientific understanding in the AI for Science era, supporting the development of scientific foundation models and a wide range of downstream scientific intelligence applications. The dataset is publicly available at https://huggingface.co/datasets/ScienceOne-AI/S1-MMAlign.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 42 references that are in the map. First posted on arXiv 1 Jan 2026.

Builds on

24 of 42 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
12 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  3. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  6. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  7. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  8. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  9. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  10. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  11. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  12. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding
  2. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
  3. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
  4. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions
  5. 2026MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph
  6. 2024Every Part Matters: Integrity Verification of Scientific Figures Based on Multimodal Large Language Models
  7. 2025Scaling Large Vision-Language Models for Enhanced Multimodal Comprehension In Biomedical Image Analysis
  8. 2025Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learning