# Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models

arXiv: https://arxiv.org/abs/2009.05175 · HTML: https://mlsanity.com/paper/2009.05175/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2020.

## Built on by (most cited first, 2 of 2)

- [Altogether: Image Captioning via Re-aligning Alt-text](https://mlsanity.com/paper/2410.17251/) (2024)
- [Curriculum Script Distillation for Multilingual Visual Question Answering](https://mlsanity.com/paper/2301.07227/) (2023)

## Abstract (arXiv)

Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples, gathered from the wild, often from noisy alt-text data. However, recent modeling approaches to IC often fall short in terms of performance in this case, because they assume a clean annotated dataset (as opposed to the noisier alt-text--based annotations), and employ an end-to-end generation approach, which often lacks both controllability and interpretability. We address these problems by breaking down the task into two simpler, more controllable tasks -- skeleton prediction and skeleton-based caption generation. Specifically, we show that selecting content words as skeletons} helps in generating improved and denoised captions when leveraging rich yet noisy alt-text--based uncurated datasets. We also show that the predicted English skeletons can be further cross-lingually leveraged to generate non-English captions, and present experimental results covering caption generation in French, Italian, German, Spanish and Hindi. We also show that skeleton-based prediction allows for better control of certain caption properties, such as length, content, and gender expression, providing a handle to perform human-in-the-loop semi-automatic corrections.