mlsanityData to 7 Oct 2026

Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023

arXiv 2501.19353 ↗First posted 31 Jan 2025cs.CL cs.AI cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Since the SciCap datasets launch in 2021, the research community has made significant progress in generating captions for scientific figures in scholarly articles. In 2023, the first SciCap Challenge took place, inviting global teams to use an expanded SciCap dataset to develop models for captioning diverse figure types across various academic fields. At the same time, text generation models advanced quickly, with many powerful pre-trained large multimodal models (LMMs) emerging that showed impressive capabilities in various vision-and-language tasks. This paper presents an overview of the first SciCap Challenge and details the performance of various models on its data, capturing a snapshot of the fields state. We found that professional editors overwhelmingly preferred figure captions generated by GPT-4V over those from all other models and even the original captions written by authors. Following this key finding, we conducted detailed analyses to answer this question: Have advanced LMMs solved the task of generating captions for scientific figures?

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 64 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

38 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov

Other references · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023Mistral 7Bcited by 3,065
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  10. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  11. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  12. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
25 more
  1. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  2. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  3. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  4. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  5. 2023The Semantic Scholar Open Data Platformcited by 109
  6. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  7. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  8. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  9. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  10. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmarkcited by 41
  11. 2021Transparent Human Evaluation for Image Captioningcited by 40
  12. 2019EvalAI: Towards Better Evaluation Systems for AI Agentscited by 34
  13. 2023ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summariescited by 30
  14. 2023SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioningcited by 24
  15. 2023GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captionscited by 21
  16. 2023The ACL OCL Corpus: Advancing Open Science in Computational Linguisticscited by 16
  17. 2023Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarizationcited by 14
  18. 2017A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networkscited by 10
  19. 2023ACL-Fig: A Dataset for Scientific Figure Classificationcited by 10
  20. 2025Multi-LLM Collaborative Caption Generation in Scientific Documentscited by 6
  21. 2024SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratingscited by 5
  22. 2025Understanding How Paper Writers Use AI-Generated Captions in Figure Caption Writingcited by 4
  23. 2024The Solution for the ICCV 2023 1st Scientific Figure Captioning Challengecited by 3
  24. 2016Iterative Aggregation Method for Solving Principal Component Analysis Problemscited by 1
  25. 2019ClangJIT: Enhancing C++ with Just-in-Time Compilationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profilescited by 2
  2. 2025Leveraging Author-Specific Context for Scientific Figure Caption Generation: 3rd SciCap Challengecited by 1
  3. 2025Five Years of SciCap: What We Learned and Future Directions for Scientific Figure Captioningcited by 0
  4. 2026MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literaturecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
  2. 2023CapsFusion: Rethinking Image-Text Data at Scale
  3. 2023FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback
  4. 2023Improving Multimodal Datasets with Image Captioning
  5. 2026S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understanding
  6. 2025Personalized Scientific Figure Caption Generation: An Empirical Study on Author-Specific Writing Style Transfer
  7. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation
  8. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions