# TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning

arXiv: https://arxiv.org/abs/2106.10936 · HTML: https://mlsanity.com/paper/2106.10936/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 21 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [Semantic Compositional Networks for Visual Captioning](https://mlsanity.com/paper/1611.08002/) (2016)
- [Hierarchy Parsing for Image Captioning](https://mlsanity.com/paper/1909.03918/) (2019)
- [Deep Reinforcement Learning-based Image Captioning with Embedding Reward](https://mlsanity.com/paper/1704.03899/) (2017)
- [Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations](https://mlsanity.com/paper/1905.06139/) (2019)

## Built on by (most cited first, 8 of 8)

- [GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features](https://mlsanity.com/paper/2207.09666/) (2022)
- [Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching](https://mlsanity.com/paper/2201.06686/) (2022)
- [Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval](https://mlsanity.com/paper/2109.05523/) (2021)
- [Image Captioning via Dynamic Path Customization](https://mlsanity.com/paper/2406.00334/) (2024)
- [Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval](https://mlsanity.com/paper/2111.03349/) (2021)
- [Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning](https://mlsanity.com/paper/2307.09915/) (2023)
- [ClickTrack: Towards Real-time Interactive Single Object Tracking](https://mlsanity.com/paper/2411.13183/) (2024)
- [Machine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments](https://mlsanity.com/paper/2605.24020/) (2026)

## Abstract (arXiv)

Existing research for image captioning usually represents an image using a scene graph with low-level facts (objects and relations) and fails to capture the high-level semantics. In this paper, we propose a Theme Concepts extended Image Captioning (TCIC) framework that incorporates theme concepts to represent high-level cross-modality semantics. In practice, we model theme concepts as memory vectors and propose Transformer with Theme Nodes (TTN) to incorporate those vectors for image captioning. Considering that theme concepts can be learned from both images and captions, we propose two settings for their representations learning based on TTN. On the vision side, TTN is configured to take both scene graph based features and theme concepts as input for visual representation learning. On the language side, TTN is configured to take both captions and theme concepts as input for text representation re-construction. Both settings aim to generate target captions with the same transformer-based decoder. During the training, we further align representations of theme concepts learned from images and corresponding captions to enforce the cross-modality learning. Experimental results on MS COCO show the effectiveness of our approach compared to some state-of-the-art models.