# A Baseline for Detecting Out-of-Distribution Examples in Image Captioning

arXiv: https://arxiv.org/abs/2207.05418 · HTML: https://mlsanity.com/paper/2207.05418/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [Neural Baby Talk](https://mlsanity.com/paper/1803.09845/) (2018)
- [Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network](https://mlsanity.com/paper/2012.07061/) (2020)
- [OODformer: Out-Of-Distribution Detection Transformer](https://mlsanity.com/paper/2107.08976/) (2021)
- [Out-of-Distribution Detection using Multiple Semantic Label Representations](https://mlsanity.com/paper/1808.06664/) (2018)
- [Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge](https://mlsanity.com/paper/2012.11696/) (2020)
- [Rich Image Captioning in the Wild](https://mlsanity.com/paper/1603.09016/) (2016)
- [Self-Supervised Out-of-Distribution Detection in Brain CT Scans](https://mlsanity.com/paper/2011.05428/) (2020)
- [Actigraphy-based Sleep/Wake Pattern Detection using Convolutional Neural Networks](https://mlsanity.com/paper/1802.07945/) (2018)
- [Generating Diverse and Informative Natural Language Fashion Feedback](https://mlsanity.com/paper/1906.06619/) (2019)
- [Joint Visual-Textual Embedding for Multimodal Style Search](https://mlsanity.com/paper/1906.06620/) (2019)
- [Redesigning the classification layer by randomizing the class representation vectors](https://mlsanity.com/paper/2011.08704/) (2020)

## Built on by (most cited first, 4 of 4)

- [Uncertainty quantification for improving radiomic-based models in radiation pneumonitis prediction](https://mlsanity.com/paper/2412.19511/) (2024)
- [Disentangling Fine-Tuning from Pre-Training in Visual Captioning with Hybrid Markov Logic](https://mlsanity.com/paper/2503.13847/) (2025)
- [On Explaining Visual Captioning with Hybrid Markov Logic Networks](https://mlsanity.com/paper/2507.21246/) (2025)
- [Simplex-Optimized Hybrid Ensemble for Large Language Model Text Detection Under Generative Distribution Drif](https://mlsanity.com/paper/2511.22153/) (2025)

## Abstract (arXiv)

Image captioning research achieved breakthroughs in recent years by developing neural models that can generate diverse and high-quality descriptions for images drawn from the same distribution as training images. However, when facing out-of-distribution (OOD) images, such as corrupted images, or images containing unknown objects, the models fail in generating relevant captions. In this paper, we consider the problem of OOD detection in image captioning. We formulate the problem and suggest an evaluation setup for assessing the model's performance on the task. Then, we analyze and show the effectiveness of the caption's likelihood score at detecting and rejecting OOD images, which implies that the relatedness between the input image and the generated caption is encapsulated within the score.