# Compositional Generalization in Image Captioning

arXiv: https://arxiv.org/abs/1909.04402 · HTML: https://mlsanity.com/paper/1909.04402/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 23 of 23)

- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning](https://mlsanity.com/paper/2203.13049/) (2022)
- [Neural-Logic Human-Object Interaction Detection](https://mlsanity.com/paper/2311.09817/) (2023)
- [Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding](https://mlsanity.com/paper/2301.09071/) (2023)
- [Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning](https://mlsanity.com/paper/2104.13872/) (2021)
- [Compositional Risk Minimization](https://mlsanity.com/paper/2410.06303/) (2024)
- [The Role of Syntactic Planning in Compositional Image Captioning](https://mlsanity.com/paper/2101.11911/) (2021)
- [A Survey on Compositional Generalization in Applications](https://mlsanity.com/paper/2302.01067/) (2023)
- [Compositional Generalization in Multilingual Semantic Parsing over Wikidata](https://mlsanity.com/paper/2108.03509/) (2021)
- [Deep soccer captioning with transformer: dataset, semantics-related losses, and multi-level evaluation](https://mlsanity.com/paper/2202.05728/) (2022)
- [SETI: Systematicity Evaluation of Textual Inference](https://mlsanity.com/paper/2305.15045/) (2023)
- [A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions](https://mlsanity.com/paper/2302.04811/) (2023)
- [MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition](https://mlsanity.com/paper/2311.01580/) (2023)
- [Resilience through Scene Context in Visual Referring Expression Generation](https://mlsanity.com/paper/2404.12289/) (2024)
- [Does Vision Accelerate Hierarchical Generalization in Neural Language Learners?](https://mlsanity.com/paper/2302.00667/) (2023)
- [Learning to generate and corr- uh I mean repair language in real-time](https://mlsanity.com/paper/2308.11683/) (2023)
- [Sequential Compositional Generalization in Multimodal Models](https://mlsanity.com/paper/2404.12013/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [KENGIC: KEyword-driven and N-Gram Graph based Image Captioning](https://mlsanity.com/paper/2302.03729/) (2023)
- [Fast and Expressive Gesture Recognition using a Combination-Homomorphic Electromyogram Encoder](https://mlsanity.com/paper/2311.14675/) (2023)
- [Attention Based Simple Primitives for Open World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2407.13715/) (2024)
- [Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs](https://mlsanity.com/paper/2509.19207/) (2025)

## Abstract (arXiv)

Image captioning models are usually evaluated on their ability to describe a held-out set of images, not on their ability to generalize to unseen concepts. We study the problem of compositional generalization, which measures how well a model composes unseen combinations of concepts when describing images. State-of-the-art image captioning models show poor generalization performance on this task. We propose a multi-task model to address the poor performance, that combines caption generation and image--sentence ranking, and uses a decoding mechanism that re-ranks the captions according their similarity to the image. This model is substantially better at generalizing to unseen combinations of concepts compared to state-of-the-art captioning models.