# Look, Read and Enrich. Learning from Scientific Figures and their Captions

arXiv: https://arxiv.org/abs/1909.09070 · HTML: https://mlsanity.com/paper/1909.09070/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 5 of 5)

- [RL-CSDia: Representation Learning of Computer Science Diagrams](https://mlsanity.com/paper/2103.05900/) (2021)
- [MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering](https://mlsanity.com/paper/2112.02839/) (2021)
- [Towards Language-driven Scientific AI](https://mlsanity.com/paper/2210.15327/) (2022)
- [Figuring out Figures: Using Textual References to Caption Scientific Figures](https://mlsanity.com/paper/2407.11008/) (2024)
- [From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding](https://mlsanity.com/paper/2609.00948/) (2026)

## Abstract (arXiv)

Compared to natural images, understanding scientific figures is particularly hard for machines. However, there is a valuable source of information in scientific literature that until now has remained untapped: the correspondence between a figure and its caption. In this paper we investigate what can be learnt by looking at a large number of figures and reading their captions, and introduce a figure-caption correspondence learning task that makes use of our observations. Training visual and language networks without supervision other than pairs of unconstrained figures and captions is shown to successfully solve this task. We also show that transferring lexical and semantic knowledge from a knowledge graph significantly enriches the resulting features. Finally, we demonstrate the positive impact of such features in other tasks involving scientific text and figures, like multi-modal classification and machine comprehension for question answering, outperforming supervised baselines and ad-hoc approaches.