# An Ensemble Model with Attention Based Mechanism for Image Captioning

arXiv: https://arxiv.org/abs/2501.14828 · HTML: https://mlsanity.com/paper/2501.14828/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 96 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [Normalized and Geometry-Aware Self-Attention Network for Image Captioning](https://mlsanity.com/paper/2003.08897/) (2020)
- [A Simple, Fast Diverse Decoding Algorithm for Neural Generation](https://mlsanity.com/paper/1611.08562/) (2016)
- [Engaging Image Captioning Via Personality](https://mlsanity.com/paper/1810.10665/) (2018)
- [Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge](https://mlsanity.com/paper/2012.12975/) (2020)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Image Captioning through Image Transformer](https://mlsanity.com/paper/2004.14231/) (2020)
- [Towards Local Visual Modeling for Image Captioning](https://mlsanity.com/paper/2302.06098/) (2023)
- [Areas of Attention for Image Captioning](https://mlsanity.com/paper/1612.01033/) (2016)
- [Multi-Modal Image Captioning for the Visually Impaired](https://mlsanity.com/paper/2105.08106/) (2021)
- [Improved Bengali Image Captioning via deep convolutional neural network based encoder-decoder model](https://mlsanity.com/paper/2102.07192/) (2021)
- [Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning](https://mlsanity.com/paper/2109.07799/) (2021)
- [Combined statistical and model based texture features for improved image classification](https://mlsanity.com/paper/1512.08814/) (2015)
- [Supervised Texture Segmentation: A Comparative Study](https://mlsanity.com/paper/1601.00212/) (2016)
- [Hyperparameter Analysis for Image Captioning](https://mlsanity.com/paper/2006.10923/) (2020)
- [Learning Style Identification Using Semi-Supervised Self-Taught Labeling](https://mlsanity.com/paper/2402.14597/) (2024)

## Built on by (most cited first, 2 of 2)

- [AGIC: Attention-Guided Image Captioning to Improve Caption Relevance](https://mlsanity.com/paper/2508.06853/) (2025)
- [Generative Retrieval for Unsupervised Text-Based Person Search](https://mlsanity.com/paper/2609.12965/) (2026)

## Abstract (arXiv)

Image captioning creates informative text from an input image by creating a relationship between the words and the actual content of an image. Recently, deep learning models that utilize transformers have been the most successful in automatically generating image captions. The capabilities of transformer networks have led to notable progress in several activities related to vision. In this paper, we thoroughly examine transformer models, emphasizing the critical role that attention mechanisms play. The proposed model uses a transformer encoder-decoder architecture to create textual captions and a deep learning convolutional neural network to extract features from the images. To create the captions, we present a novel ensemble learning framework that improves the richness of the generated captions by utilizing several deep neural network architectures based on a voting mechanism that chooses the caption with the highest bilingual evaluation understudy (BLEU) score. The proposed model was evaluated using publicly available datasets. Using the Flickr8K dataset, the proposed model achieved the highest BLEU-[1-3] scores with rates of 0.728, 0.495, and 0.323, respectively. The suggested model outperformed the latest methods in Flickr30k datasets, determined by BLEU-[1-4] scores with rates of 0.798, 0.561, 0.387, and 0.269, respectively. The model efficacy was also obtained by the Semantic propositional image caption evaluation (SPICE) metric with a scoring rate of 0.164 for the Flicker8k dataset and 0.387 for the Flicker30k. Finally, ensemble learning significantly advances the process of image captioning and, hence, can be leveraged in various applications across different domains.