# Learning to Collocate Neural Modules for Image Captioning

arXiv: https://arxiv.org/abs/1904.08608 · HTML: https://mlsanity.com/paper/1904.08608/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2019.

## Built on by (most cited first, 22 of 22)

- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Text-Only Training for Image Captioning using Noise-Injected CLIP](https://mlsanity.com/paper/2211.00575/) (2022)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features](https://mlsanity.com/paper/2207.09666/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Unifying Multimodal Transformer for Bi-directional Image and Text Generation](https://mlsanity.com/paper/2110.09753/) (2021)
- [Retrieval-Augmented Transformer for Image Captioning](https://mlsanity.com/paper/2207.13162/) (2022)
- [Auto-Parsing Network for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2108.10568/) (2021)
- [Transforming Visual Scene Graphs to Image Captions](https://mlsanity.com/paper/2305.02177/) (2023)
- [MMNet: Multi-Collaboration and Multi-Supervision Network for Sequential Deepfake Detection](https://mlsanity.com/paper/2307.02733/) (2023)
- [CaMEL: Mean Teacher Learning for Image Captioning](https://mlsanity.com/paper/2202.10492/) (2022)
- [Truth-Conditional Captioning of Time Series Data](https://mlsanity.com/paper/2110.01839/) (2021)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping](https://mlsanity.com/paper/2304.13273/) (2023)
- [Reverse Stable Diffusion: What prompt was used to generate this image?](https://mlsanity.com/paper/2308.01472/) (2023)
- [Image Captioning In the Transformer Age](https://mlsanity.com/paper/2204.07374/) (2022)
- [Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning](https://mlsanity.com/paper/2210.01338/) (2022)
- [Macroscopic Control of Text Generation for Image Captioning](https://mlsanity.com/paper/2101.08000/) (2021)
- [A Comprehensive Analysis of Real-World Image Captioning and Scene Identification](https://mlsanity.com/paper/2308.02833/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)

## Abstract (arXiv)

We do not speak word by word from scratch; our brain quickly structures a pattern like textscsth do sth at someplace and then fill in the detailed descriptions. To render existing encoder-decoder image captioners such human-like reasoning, we propose a novel framework: learning to Collocate Neural Modules (CNM), to generate the `inner pattern' connecting visual encoder and language decoder. Unlike the widely-used neural module networks in visual Q&A, where the language (ie, question) is fully observable, CNM for captioning is more challenging as the language is being generated and thus is partially observable. To this end, we make the following technical contributions for CNM training: 1) compact module design --- one for function words and three for visual content words (eg, noun, adjective, and verb), 2) soft module fusion and multi-step module execution, robustifying the visual reasoning in partial observation, 3) a linguistic loss for module controller being faithful to part-of-speech collocations (eg, adjective is before noun). Extensive experiments on the challenging MS-COCO image captioning benchmark validate the effectiveness of our CNM image captioner. In particular, CNM achieves a new state-of-the-art 127.9 CIDEr-D on Karpathy split and a single-model 126.0 c40 on the official server. CNM is also robust to few training samples, eg, by training only one sentence per image, CNM can halve the performance loss compared to a strong baseline.