mlsanityData to 14 Oct 2026

SciCap: Generating Captions for Scientific Figures

arXiv 2110.11624 ↗First posted 22 Oct 2021cs.CL cs.AI cs.CVcited by 77

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Researchers use figures to communicate rich, complex information in scientific papers. The captions of these figures are critical to conveying effective messages. However, low-quality figure captions commonly occur in scientific articles and may decrease understanding. In this paper, we propose an end-to-end neural framework to automatically generate informative, high-quality captions for scientific figures. To this end, we introduce SCICAP, a large-scale figure-caption dataset based on computer science arXiv papers published between 2010 and 2020. After pre-processing - including figure-type classification, sub-figure identification, text normalization, and caption text selection - SCICAP contained more than two million figures extracted from over 290,000 papers. We then established baseline models that caption graph plots, the dominant (19.2%) figure type. The experimental results showed both opportunities and steep challenges of generating captions for scientific figures.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 22 Oct 2021.

Builds on

17 of 34 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  5. 2014How transferable are features in deep neural networks?cited by 1,225
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  7. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  8. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  9. 2019On the Use of ArXiv as a Datasetcited by 103
  10. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 78
  11. 2021Quantifying Bias in Automatic Speech Recognitioncited by 50
  12. 2019Figure Captioning with Reasoning and Sequence-Level Trainingcited by 21
5 more
  1. 2019MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehensioncited by 20
  2. 2017A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networkscited by 10
  3. 2015Building Proteins in a Day: Efficient 3D Molecular Reconstructioncited by 3
  4. 2020ScanSSD: Scanning Single Shot Detector for Mathematical Formulas in PDF Document Imagescited by 3
  5. 2017Incremental DFS algorithms: a theoretical and experimental studycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 17; 2024: 18; 2025: 18; 2026: 20.

77 papers.

Sort:

By area:

  1. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  2. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  3. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  4. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  5. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  6. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
  7. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 92
  8. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 79
  9. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  10. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  11. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  12. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
38 more
  1. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 48
  2. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 46
  3. 2023ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summariescited by 32
  4. 2023mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Modelcited by 32
  5. 2022OCR-VQGAN: Taming Text-within-Image Generationcited by 31
  6. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 30
  7. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructionscited by 27
  8. 2024FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understandingcited by 27
  9. 2023SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioningcited by 24
  10. 2023GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captionscited by 21
  11. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructionscited by 19
  12. 2022LineCap: Line Charts for Data Visualization Captioning Modelscited by 17
  13. 2023The ACL OCL Corpus: Advancing Open Science in Computational Linguisticscited by 17
  14. 2023Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairscited by 15
  15. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  16. 2023Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarizationcited by 14
  17. 2023PatFig: Generating Short and Long Captions for Patent Figurescited by 12
  18. 2024cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Paperscited by 11
  19. 2025Charts Are Not Images: On the Challenges of Scientific Chart Editingcited by 11
  20. 2026QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understandingcited by 11
  21. 2022TaTa: A Multilingual Table-to-Text Dataset for African Languagescited by 10
  22. 2025Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterizationcited by 9
  23. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  24. 2025MuSciClaims: Multimodal Scientific Claim Verificationcited by 8
  25. 2023FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedbackcited by 7
  26. 2025PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figurescited by 7
  27. 2025A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Toolscited by 7
  28. 2025ChartCap: Mitigating Hallucination of Dense Chart Captioningcited by 7
  29. 2023Tackling Hallucinations in Neural Chart Summarizationcited by 6
  30. 2024Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directionscited by 6
  31. 2025Multi-LLM Collaborative Caption Generation in Scientific Documentscited by 6
  32. 2025Chart-HQA: A Benchmark for Hypothetical Question Answering in Chartscited by 6
  33. 2024SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratingscited by 5
  34. 2024Faithful Chart Summarization with ChaTS-Picited by 5
  35. 2026OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understandingcited by 5
  36. 2025Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023cited by 4
  37. 2025Exploring LLMs for Scientific Information Extraction Using The SciEx Frameworkcited by 4
  38. 2026SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detectioncited by 4

50 most cited of 77 All 77 (JSON)