# Figure Captioning with Reasoning and Sequence-Level Training

arXiv: https://arxiv.org/abs/1906.02850 · HTML: https://mlsanity.com/paper/1906.02850/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2019.

## Built on by (most cited first, 21 of 21)

- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://mlsanity.com/paper/2403.00231/) (2024)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ](https://mlsanity.com/paper/2310.00367/) (2023)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models](https://mlsanity.com/paper/2403.12027/) (2024)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [OCR-VQGAN: Taming Text-within-Image Generation](https://mlsanity.com/paper/2210.11248/) (2022)
- [ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules](https://mlsanity.com/paper/2304.02173/) (2023)
- [SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning](https://mlsanity.com/paper/2306.03491/) (2023)
- [PatFig: Generating Short and Long Captions for Patent Figures](https://mlsanity.com/paper/2309.08379/) (2023)
- [Truth-Conditional Captioning of Time Series Data](https://mlsanity.com/paper/2110.01839/) (2021)
- [Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions](https://mlsanity.com/paper/2409.19747/) (2024)
- [GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models](https://mlsanity.com/paper/2408.11817/) (2024)
- [TADACap: Time-series Adaptive Domain-Aware Captioning](https://mlsanity.com/paper/2504.11441/) (2025)
- [GoT-CQA: Graph-of-Thought Guided Compositional Reasoning for Chart Question Answering](https://mlsanity.com/paper/2409.02611/) (2024)
- [GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models](https://mlsanity.com/paper/2604.04172/) (2026)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [Enhancing Scientific Figure Captioning Through Cross-modal Learning](https://mlsanity.com/paper/2406.17047/) (2024)
- [Text2Arch: A Dataset for Generating Scientific Architecture Diagrams from Natural Language Descriptions](https://mlsanity.com/paper/2604.14941/) (2026)

## Abstract (arXiv)

Figures, such as bar charts, pie charts, and line plots, are widely used to convey important information in a concise format. They are usually human-friendly but difficult for computers to process automatically. In this work, we investigate the problem of figure captioning where the goal is to automatically generate a natural language description of the figure. While natural image captioning has been studied extensively, figure captioning has received relatively little attention and remains a challenging problem. First, we introduce a new dataset for figure captioning, FigCAP, based on FigureQA. Second, we propose two novel attention mechanisms. To achieve accurate generation of labels in figures, we propose Label Maps Attention. To model the relations between figure labels, we propose Relation Maps Attention. Third, we use sequence-level training with reinforcement learning in order to directly optimizes evaluation metrics, which alleviates the exposure bias issue and further improves the models in generating long captions. Extensive experiments show that the proposed method outperforms the baselines, thus demonstrating a significant potential for the automatic captioning of vast repositories of figures.