# Aligning Linguistic Words and Visual Semantic Units for Image Captioning

arXiv: https://arxiv.org/abs/1908.02127 · HTML: https://mlsanity.com/paper/1908.02127/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2019.

## Built on by (most cited first, 25 of 25)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal](https://mlsanity.com/paper/2212.05767/) (2022)
- [Unifying Multimodal Transformer for Bi-directional Image and Text Generation](https://mlsanity.com/paper/2110.09753/) (2021)
- [Domain Adaptive Semantic Segmentation without Source Data](https://mlsanity.com/paper/2110.06484/) (2021)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment](https://mlsanity.com/paper/2212.10549/) (2022)
- [OSCAR-Net: Object-centric Scene Graph Attention for Image Attribution](https://mlsanity.com/paper/2108.03541/) (2021)
- [X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering](https://mlsanity.com/paper/2107.11576/) (2021)
- [Graph Neural Networks in Vision-Language Image Understanding: A Survey](https://mlsanity.com/paper/2303.03761/) (2023)
- [Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning](https://mlsanity.com/paper/2108.11912/) (2021)
- [Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets](https://mlsanity.com/paper/2111.12727/) (2021)
- [Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch](https://mlsanity.com/paper/2108.07353/) (2021)
- [SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning](https://mlsanity.com/paper/2512.01975/) (2025)
- [Geometry-Entangled Visual Semantic Transformer for Image Captioning](https://mlsanity.com/paper/2109.14137/) (2021)
- [Semantic Composition in Visually Grounded Language Models](https://mlsanity.com/paper/2305.16328/) (2023)
- [A Comprehensive Analysis of Real-World Image Captioning and Scene Identification](https://mlsanity.com/paper/2308.02833/) (2023)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Controllable Image Captioning](https://mlsanity.com/paper/2204.13324/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)

## Abstract (arXiv)

Image captioning attempts to generate a sentence composed of several linguistic words, which are used to describe objects, attributes, and interactions in an image, denoted as visual semantic units in this paper. Based on this view, we propose to explicitly model the object interactions in semantics and geometry based on Graph Convolutional Networks (GCNs), and fully exploit the alignment between linguistic words and visual semantic units for image captioning. Particularly, we construct a semantic graph and a geometry graph, where each node corresponds to a visual semantic unit, i.e., an object, an attribute, or a semantic (geometrical) interaction between two objects. Accordingly, the semantic (geometrical) context-aware embeddings for each unit are obtained through the corresponding GCN learning processers. At each time step, a context gated attention module takes as inputs the embeddings of the visual semantic units and hierarchically align the current word with these units by first deciding which type of visual semantic unit (object, attribute, or interaction) the current word is about, and then finding the most correlated visual semantic units under this type. Extensive experiments are conducted on the challenging MS-COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches.