# A Comprehensive Analysis of Real-World Image Captioning and Scene Identification

arXiv: https://arxiv.org/abs/2308.02833 · HTML: https://mlsanity.com/paper/2308.02833/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 5 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions](https://mlsanity.com/paper/2303.06594/) (2023)
- [Improving Image Captioning with Better Use of Captions](https://mlsanity.com/paper/2006.11807/) (2020)
- [CPTR: Full Transformer Network for Image Captioning](https://mlsanity.com/paper/2101.10804/) (2021)
- [Hierarchy Parsing for Image Captioning](https://mlsanity.com/paper/1909.03918/) (2019)
- [Context-Aware Visual Policy Network for Fine-Grained Image Captioning](https://mlsanity.com/paper/1906.02365/) (2019)
- [Show, Recall, and Tell: Image Captioning with Recall Mechanism](https://mlsanity.com/paper/2001.05876/) (2020)
- [Aligning Linguistic Words and Visual Semantic Units for Image Captioning](https://mlsanity.com/paper/1908.02127/) (2019)
- [Learning to Collocate Neural Modules for Image Captioning](https://mlsanity.com/paper/1904.08608/) (2019)
- [IC3: Image Captioning by Committee Consensus](https://mlsanity.com/paper/2302.01328/) (2023)
- [Explain Images with Multimodal Recurrent Neural Networks](https://mlsanity.com/paper/1410.1090/) (2014)
- [Adaptively Aligned Image Captioning via Adaptive Attention Time](https://mlsanity.com/paper/1909.09060/) (2019)
- [Image Captioning In the Transformer Age](https://mlsanity.com/paper/2204.07374/) (2022)

## Built on by (most cited first, 1 of 1)

- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)

## Abstract (arXiv)

Image captioning is a computer vision task that involves generating natural language descriptions for images. This method has numerous applications in various domains, including image retrieval systems, medicine, and various industries. However, while there has been significant research in image captioning, most studies have focused on high quality images or controlled environments, without exploring the challenges of real-world image captioning. Real-world image captioning involves complex and dynamic environments with numerous points of attention, with images which are often very poor in quality, making it a challenging task, even for humans. This paper evaluates the performance of various models that are built on top of different encoding mechanisms, language decoders and training procedures using a newly created real-world dataset that consists of over 800+ images of over 65 different scene classes, built using MIT Indoor scenes dataset. This dataset is captioned using the IC3 approach that generates more descriptive captions by summarizing the details that are covered by standard image captioning models from unique view-points of the image.