# Image Captioning through Image Transformer

arXiv: https://arxiv.org/abs/2004.14231 · HTML: https://mlsanity.com/paper/2004.14231/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 29 of 29)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [β-Variational autoencoders and transformers for reduced-order modelling of fluid flows](https://mlsanity.com/paper/2304.03571/) (2023)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning](https://mlsanity.com/paper/2108.02366/) (2021)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Iterative Scene Graph Generation](https://mlsanity.com/paper/2207.13440/) (2022)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning](https://mlsanity.com/paper/2212.13563/) (2022)
- [M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning](https://mlsanity.com/paper/2409.15657/) (2024)
- [Journalistic Guidelines Aware News Image Captioning](https://mlsanity.com/paper/2109.02865/) (2021)
- [BodyMap: Learning Full-Body Dense Correspondence Map](https://mlsanity.com/paper/2205.09111/) (2022)
- [Graph Neural Networks in Vision-Language Image Understanding: A Survey](https://mlsanity.com/paper/2303.03761/) (2023)
- [Bornon: Bengali Image Captioning with Transformer-based Deep learning approach](https://mlsanity.com/paper/2109.05218/) (2021)
- [Channel-Partitioned Windowed Attention And Frequency Learning for Single Image Super-Resolution](https://mlsanity.com/paper/2407.16232/) (2024)
- [A Spatio-Temporal Attentive Network for Video-Based Crowd Counting](https://mlsanity.com/paper/2208.11339/) (2022)
- [Explaining Face Presentation Attack Detection Using Natural Language](https://mlsanity.com/paper/2111.04862/) (2021)
- [Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation](https://mlsanity.com/paper/2112.14088/) (2021)
- [An Ensemble Model with Attention Based Mechanism for Image Captioning](https://mlsanity.com/paper/2501.14828/) (2025)
- [L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training](https://mlsanity.com/paper/2507.08710/) (2025)
- [The impact of memory on learning sequence-to-sequence tasks](https://mlsanity.com/paper/2205.14683/) (2022)
- [Towards Universal Vision-language Omni-supervised Segmentation](https://mlsanity.com/paper/2303.06547/) (2023)
- [Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation](https://mlsanity.com/paper/2506.05399/) (2025)
- [LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation](https://mlsanity.com/paper/2109.04993/) (2021)
- [Technical Language Supervision for Intelligent Fault Diagnosis in Process Industry](https://mlsanity.com/paper/2112.07356/) (2021)
- [Controllable Image Captioning](https://mlsanity.com/paper/2204.13324/) (2022)
- [VieCap4H-VLSP 2021: ObjectAoA-Enhancing performance of Object Relation Transformer with Attention on Attention for Vietnamese image captioning](https://mlsanity.com/paper/2211.05405/) (2022)
- [ViTOC: Vision Transformer and Object-aware Captioner](https://mlsanity.com/paper/2411.07265/) (2024)
- [From Image Captioning to Visual Storytelling](https://mlsanity.com/paper/2508.14045/) (2025)

## Abstract (arXiv)

Automatic captioning of images is a task that combines the challenges of image analysis and text generation. One important aspect in captioning is the notion of attention: How to decide what to describe and in which order. Inspired by the successes in text analysis and translation, previous work have proposed the textittransformer architecture for image captioning. However, the structure between the textitsemantic units in images (usually the detected regions from object detection model) and sentences (each single word) is different. Limited work has been done to adapt the transformer's internal architecture to images. In this work, we introduce the textbftextitimage transformer, which consists of a modified encoding transformer and an implicit decoding transformer, motivated by the relative spatial relationship between image regions. Our design widen the original transformer layer's inner architecture to adapt to the structure of images. With only regions feature as inputs, our model achieves new state-of-the-art performance on both MSCOCO offline and online testing benchmarks.