# Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network

arXiv: https://arxiv.org/abs/2012.07061 · HTML: https://mlsanity.com/paper/2012.07061/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 30 of 48)

- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation](https://mlsanity.com/paper/2308.12604/) (2023)
- [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://mlsanity.com/paper/2308.05095/) (2023)
- [KiUT: Knowledge-injected U-Transformer for Radiology Report Generation](https://mlsanity.com/paper/2306.11345/) (2023)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features](https://mlsanity.com/paper/2207.09666/) (2022)
- [Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing](https://mlsanity.com/paper/2412.19806/) (2024)
- [Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2204.10688/) (2022)
- [End-to-End Transformer Based Model for Image Captioning](https://mlsanity.com/paper/2203.15350/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning](https://mlsanity.com/paper/2205.04363/) (2022)
- [CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation](https://mlsanity.com/paper/2211.01412/) (2022)
- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement](https://mlsanity.com/paper/2208.08965/) (2022)
- [Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning](https://mlsanity.com/paper/2208.06551/) (2022)
- [Controllable Image Captioning via Prompting](https://mlsanity.com/paper/2212.01803/) (2022)
- [Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment](https://mlsanity.com/paper/2305.12260/) (2023)
- [Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling](https://mlsanity.com/paper/2308.05081/) (2023)
- [Efficient Image Captioning for Edge Devices](https://mlsanity.com/paper/2212.08985/) (2022)
- [Multi-Granular Multimodal Clue Fusion for Meme Understanding](https://mlsanity.com/paper/2503.12560/) (2025)
- [Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Network](https://mlsanity.com/paper/2301.03160/) (2023)
- [Show, Interpret and Tell: Entity-aware Contextualised Image Captioning in Wikipedia](https://mlsanity.com/paper/2209.10474/) (2022)
- [HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning](https://mlsanity.com/paper/2305.16295/) (2023)
- [Image Captioning via Compact Bidirectional Architecture](https://mlsanity.com/paper/2201.01984/) (2022)
- [Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning](https://mlsanity.com/paper/2309.04965/) (2023)
- [Efficient Modeling of Future Context for Image Captioning](https://mlsanity.com/paper/2207.10897/) (2022)
- [A Baseline for Detecting Out-of-Distribution Examples in Image Captioning](https://mlsanity.com/paper/2207.05418/) (2022)

## Abstract (arXiv)

Transformer-based architectures have shown great success in image captioning, where object regions are encoded and then attended into the vectorial representations to guide the caption decoding. However, such vectorial representations only contain region-level information without considering the global information reflecting the entire image, which fails to expand the capability of complex multi-modal reasoning in image captioning. In this paper, we introduce a Global Enhanced Transformer (termed GET) to enable the extraction of a more comprehensive global representation, and then adaptively guide the decoder to generate high-quality captions. In GET, a Global Enhanced Encoder is designed for the embedding of the global feature, and a Global Adaptive Decoder are designed for the guidance of the caption generation. The former models intra- and inter-layer global representation by taking advantage of the proposed Global Enhanced Attention and a layer-wise fusion module. The latter contains a Global Adaptive Controller that can adaptively fuse the global information into the decoder to guide the caption generation. Extensive experiments on MS COCO dataset demonstrate the superiority of our GET over many state-of-the-arts.