# Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents

arXiv: https://arxiv.org/abs/1904.07826 · HTML: https://mlsanity.com/paper/1904.07826/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2019.

## Built on by (most cited first, 9 of 9)

- [Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text](https://mlsanity.com/paper/2304.06939/) (2023)
- [VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer](https://mlsanity.com/paper/2107.02681/) (2021)
- [Semantic Alignment for Multimodal Large Language Models](https://mlsanity.com/paper/2408.12867/) (2024)
- [Weakly Supervised Face Naming with Symmetry-Enhanced Contrastive Loss](https://mlsanity.com/paper/2210.08957/) (2022)
- [Latent Alignment of Procedural Concepts in Multimodal Recipes](https://mlsanity.com/paper/2101.04727/) (2021)
- [An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information](https://mlsanity.com/paper/2104.02605/) (2021)
- [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](https://mlsanity.com/paper/2204.02566/) (2022)
- [Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification](https://mlsanity.com/paper/2303.15016/) (2023)
- [Are Large Language Models Good Data Preprocessors?](https://mlsanity.com/paper/2502.16790/) (2025)

## Abstract (arXiv)

Images and text co-occur constantly on the web, but explicit links between images and sentences (or other intra-document textual units) are often not present. We present algorithms that discover image-sentence relationships without relying on explicit multimodal annotation in training. We experiment on seven datasets of varying difficulty, ranging from documents consisting of groups of images captioned post hoc by crowdworkers to naturally-occurring user-generated multimodal documents. We find that a structured training objective based on identifying whether collections of images and sentences co-occur in documents can suffice to predict links between specific sentences and specific images within the same document at test time.