# Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization

arXiv: https://arxiv.org/abs/2302.12324 · HTML: https://mlsanity.com/paper/2302.12324/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 67 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Logical Natural Language Generation from Open-Domain Tables](https://mlsanity.com/paper/2004.10404/) (2020)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Plan-then-Generate: Controlled Data-to-Text Generation via Planning](https://mlsanity.com/paper/2108.13740/) (2021)
- [Logic2Text: High-Fidelity Natural Language Generation from Logical Forms](https://mlsanity.com/paper/2004.14579/) (2020)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [LineCap: Line Charts for Data Visualization Captioning Models](https://mlsanity.com/paper/2207.07243/) (2022)
- [AutoChart: A Dataset for Chart-to-Text Generation Task](https://mlsanity.com/paper/2108.06897/) (2021)
- [Why More Text is (Often) Better: Themes from Reader Preferences for Integration of Charts and Text](https://mlsanity.com/paper/2209.10789/) (2022)
- [Data-selective Transfer Learning for Multi-Domain Speech Recognition](https://mlsanity.com/paper/1509.02409/) (2015)
- [Theme-aware generation model for chinese lyrics](https://mlsanity.com/paper/1906.02134/) (2019)
- [Semantic Frame Forecast](https://mlsanity.com/paper/2104.05604/) (2021)
- [DialogAct2Vec: Towards End-to-End Dialogue Agent by Multi-Task Representation Learning](https://mlsanity.com/paper/1911.04088/) (2019)

## Built on by (most cited first, 14 of 14)

- [AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ](https://mlsanity.com/paper/2310.00367/) (2023)
- [GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions](https://mlsanity.com/paper/2310.15405/) (2023)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback](https://mlsanity.com/paper/2307.10867/) (2023)
- [No perspective, no perception!! Perspective-aware Healthcare Answer Summarization](https://mlsanity.com/paper/2406.08881/) (2024)
- [Multi-LLM Collaborative Caption Generation in Scientific Documents](https://mlsanity.com/paper/2501.02552/) (2025)
- [SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratings](https://mlsanity.com/paper/2403.17784/) (2024)
- [Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023](https://mlsanity.com/paper/2501.19353/) (2025)
- [GoT-CQA: Graph-of-Thought Guided Compositional Reasoning for Chart Question Answering](https://mlsanity.com/paper/2409.02611/) (2024)
- [LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles](https://mlsanity.com/paper/2506.06561/) (2025)
- [Enhancing Scientific Figure Captioning Through Cross-modal Learning](https://mlsanity.com/paper/2406.17047/) (2024)
- [Harnessing PDF Data for Improving Japanese Large Multimodal Models](https://mlsanity.com/paper/2502.14778/) (2025)
- [Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models](https://mlsanity.com/paper/2603.02865/) (2026)
- [Personalized Scientific Figure Caption Generation: An Empirical Study on Author-Specific Writing Style Transfer](https://mlsanity.com/paper/2509.25817/) (2025)

## Abstract (arXiv)

Good figure captions help paper readers understand complex scientific figures. Unfortunately, even published papers often have poorly written captions. Automatic caption generation could aid paper writers by providing good starting captions that can be refined for better quality. Prior work often treated figure caption generation as a vision-to-language task. In this paper, we show that it can be more effectively tackled as a text summarization task in scientific documents. We fine-tuned PEGASUS, a pre-trained abstractive summarization model, to specifically summarize figure-referencing paragraphs (e.g., "Figure 3 shows...") into figure captions. Experiments on large-scale arXiv figures show that our method outperforms prior vision methods in both automatic and human evaluations. We further conducted an in-depth investigation focused on two key challenges: (i) the common presence of low-quality author-written captions and (ii) the lack of clear standards for good captions. Our code and data are available at: https://github.com/Crowd-AI-Lab/Generating-Figure-Captions-as-a-Text-Summarization-Task.