# End-to-End Transformer Based Model for Image Captioning

arXiv: https://arxiv.org/abs/2203.15350 · HTML: https://mlsanity.com/paper/2203.15350/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [Dual-Level Collaborative Transformer for Image Captioning](https://mlsanity.com/paper/2101.06462/) (2021)
- [Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network](https://mlsanity.com/paper/2012.07061/) (2020)
- [Recurrent Fusion Network for Image Captioning](https://mlsanity.com/paper/1807.09986/) (2018)

## Built on by (most cited first, 30 of 38)

- [PRIOR: Prototype Representation Joint Learning from Medical Images and Reports](https://mlsanity.com/paper/2307.12577/) (2023)
- [Tuning computer vision models with task rewards](https://mlsanity.com/paper/2302.08242/) (2023)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning](https://mlsanity.com/paper/2208.06551/) (2022)
- [Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning](https://mlsanity.com/paper/2312.01191/) (2023)
- [Controllable Image Captioning via Prompting](https://mlsanity.com/paper/2212.01803/) (2022)
- [Transforming Visual Scene Graphs to Image Captions](https://mlsanity.com/paper/2305.02177/) (2023)
- [Efficient Image Captioning for Edge Devices](https://mlsanity.com/paper/2212.08985/) (2022)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [Time Series Language Model for Descriptive Caption Generation](https://mlsanity.com/paper/2501.01832/) (2025)
- [What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioning](https://mlsanity.com/paper/2310.20607/) (2023)
- [Image Captioning with Multi-Context Synthetic Data](https://mlsanity.com/paper/2305.18072/) (2023)
- [Cycle-Consistency Learning for Captioning and Grounding](https://mlsanity.com/paper/2312.15162/) (2023)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [Channel-Partitioned Windowed Attention And Frequency Learning for Single Image Super-Resolution](https://mlsanity.com/paper/2407.16232/) (2024)
- [GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text](https://mlsanity.com/paper/2312.15320/) (2023)
- [Progressive Tree-Structured Prototype Network for End-to-End Image Captioning](https://mlsanity.com/paper/2211.09460/) (2022)
- [Survey of Multimodal Geospatial Foundation Models: Techniques, Applications, and Challenges](https://mlsanity.com/paper/2510.22964/) (2025)
- [A request for clarity over the End of Sequence token in the Self-Critical Sequence Training](https://mlsanity.com/paper/2305.12254/) (2023)
- [Shifted Window Fourier Transform And Retention For Image Captioning](https://mlsanity.com/paper/2408.13963/) (2024)
- [Analyzing the Sensitivity of Vision Language Models in Visual Question Answering](https://mlsanity.com/paper/2507.21335/) (2025)
- [Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning](https://mlsanity.com/paper/2307.09915/) (2023)
- [Text Data-Centric Image Captioning with Interactive Prompts](https://mlsanity.com/paper/2403.19193/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation](https://mlsanity.com/paper/2506.05399/) (2025)
- [OSIC: A New One-Stage Image Captioner Coined](https://mlsanity.com/paper/2211.02321/) (2022)
- [Soft Alignment Objectives for Robust Adaptation of Language Generation](https://mlsanity.com/paper/2211.16550/) (2022)
- [Adaptively Clustering Neighbor Elements for Image-Text Generation](https://mlsanity.com/paper/2301.01955/) (2023)
- [Transformer based Multitask Learning for Image Captioning and Object Detection](https://mlsanity.com/paper/2403.06292/) (2024)

## Abstract (arXiv)

CNN-LSTM based architectures have played an important role in image captioning, but limited by the training efficiency and expression ability, researchers began to explore the CNN-Transformer based models and achieved great success. Meanwhile, almost all recent works adopt Faster R-CNN as the backbone encoder to extract region-level features from given images. However, Faster R-CNN needs a pre-training on an additional dataset, which divides the image captioning task into two stages and limits its potential applications. In this paper, we build a pure Transformer-based model, which integrates image captioning into one stage and realizes end-to-end training. Firstly, we adopt SwinTransformer to replace Faster R-CNN as the backbone encoder to extract grid-level features from given images; Then, referring to Transformer, we build a refining encoder and a decoder. The refining encoder refines the grid features by capturing the intra-relationship between them, and the decoder decodes the refined features into captions word by word. Furthermore, in order to increase the interaction between multi-modal (vision and language) features to enhance the modeling capability, we calculate the mean pooling of grid features as the global feature, then introduce it into refining encoder to refine with grid features together, and add a pre-fusion process of refined global feature and generated words in decoder. To validate the effectiveness of our proposed model, we conduct experiments on MSCOCO dataset. The experimental results compared to existing published works demonstrate that our model achieves new state-of-the-art performances of 138.2% (single model) and 141.0% (ensemble of 4 models) CIDEr scores on `Karpathy' offline test split and 136.0% (c5) and 138.3% (c40) CIDEr scores on the official online test server. Trained models and source code will be released.