# Five Years of SciCap: What We Learned and Future Directions for Scientific Figure Captioning

arXiv: https://arxiv.org/abs/2512.21789 · HTML: https://mlsanity.com/paper/2512.21789/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 25 Dec 2025.

## Builds on (most cited first, 9 of 9)

- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions](https://mlsanity.com/paper/2310.15405/) (2023)
- [SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratings](https://mlsanity.com/paper/2403.17784/) (2024)
- [Understanding How Paper Writers Use AI-Generated Captions in Figure Caption Writing](https://mlsanity.com/paper/2501.06317/) (2025)
- [Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023](https://mlsanity.com/paper/2501.19353/) (2025)

## Abstract (arXiv)

Between 2021 and 2025, the SciCap project grew from a small seed-funded idea at The Pennsylvania State University (Penn State) into one of the central efforts shaping the scientific figure-captioning landscape. Supported by a Penn State seed grant, Adobe, and the Alfred P. Sloan Foundation, what began as our attempt to test whether domain-specific training, which was successful in text models like SciBERT, could also work for figure captions expanded into a multi-institution collaboration. Over these five years, we curated, released, and continually updated a large collection of figure-caption pairs from arXiv papers, conducted extensive automatic and human evaluations on both generated and author-written captions, navigated the rapid rise of large language models (LLMs), launched annual challenges, and built interactive systems that help scientists write better captions. In this piece, we look back at the first five years of SciCap and summarize the key technical and methodological lessons we learned. We then outline five major unsolved challenges and propose directions for the next phase of research in scientific figure captioning.