# Image Captioners Sometimes Tell More Than Images They See

arXiv: https://arxiv.org/abs/2305.02932 · HTML: https://mlsanity.com/paper/2305.02932/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Generating Images from Captions with Attention](https://mlsanity.com/paper/1511.02793/) (2015)
- [CrisisMMD: Multimodal Twitter Datasets from Natural Disasters](https://mlsanity.com/paper/1805.00713/) (2018)
- [Multimodal Emotion Recognition with High-level Speech and Text Features](https://mlsanity.com/paper/2111.10202/) (2021)
- [Deep Learning Benchmarks and Datasets for Social Media Image Classification for Disaster Response](https://mlsanity.com/paper/2011.08916/) (2020)
- [Representation learning through cross-modal conditional teacher-student training for speech emotion recognition](https://mlsanity.com/paper/2112.00158/) (2021)
- [Joint Speech Recognition and Audio Captioning](https://mlsanity.com/paper/2202.01405/) (2022)

## Built on by (most cited first, 2 of 2)

- [Vulnerability of Text-to-Image Models to Prompt Template Stealing: A Differential Evolution Approach](https://mlsanity.com/paper/2502.14285/) (2025)
- [Zero-shot Hazard Identification in Autonomous Driving: A Case Study on the COOOL Benchmark](https://mlsanity.com/paper/2412.19944/) (2024)

## Abstract (arXiv)

Image captioning, a.k.a. "image-to-text," which generates descriptive text from given images, has been rapidly developing throughout the era of deep learning. To what extent is the information in the original image preserved in the descriptive text generated by an image captioner? To answer that question, we have performed experiments involving the classification of images from descriptive text alone, without referring to the images at all, and compared results with those from standard image-based classifiers. We have evaluate several image captioning models with respect to a disaster image classification task, CrisisNLP, and show that descriptive text classifiers can sometimes achieve higher accuracy than standard image-based classifiers. Further, we show that fusing an image-based classifier with a descriptive text classifier can provide improvement in accuracy.