# Rich Image Captioning in the Wild

arXiv: https://arxiv.org/abs/1603.09016 · HTML: https://mlsanity.com/paper/1603.09016/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2016.

## Built on by (most cited first, 11 of 11)

- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning](https://mlsanity.com/paper/2212.13563/) (2022)
- [Computer Vision and Conflicting Values: Describing People with Automated Alt Text](https://mlsanity.com/paper/2105.12754/) (2021)
- [Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model](https://mlsanity.com/paper/2308.01126/) (2023)
- [IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System](https://mlsanity.com/paper/2109.04202/) (2021)
- [A Baseline for Detecting Out-of-Distribution Examples in Image Captioning](https://mlsanity.com/paper/2207.05418/) (2022)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [RCA-NOC: Relative Contrastive Alignment for Novel Object Captioning](https://mlsanity.com/paper/2312.06299/) (2023)
- [Image Captioning based on Feature Refinement and Reflective Decoding](https://mlsanity.com/paper/2206.07986/) (2022)
- [Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model](https://mlsanity.com/paper/2505.23358/) (2025)
- [Leveraging GenAI for Segmenting and Labeling Centuries-old Technical Documents](https://mlsanity.com/paper/2603.00147/) (2026)

## Abstract (arXiv)

We present an image caption system that addresses new challenges of automatically describing images in the wild. The challenges include high quality caption quality with respect to human judgments, out-of-domain data handling, and low latency required in many applications. Built on top of a state-of-the-art framework, we developed a deep vision model that detects a broad range of visual concepts, an entity recognition model that identifies celebrities and landmarks, and a confidence model for the caption output. Experimental results show that our caption engine outperforms previous state-of-the-art systems significantly on both in-domain dataset (i.e. MS COCO) and out of-domain datasets.