# iCap: Interactive Image Captioning with Predictive Text

arXiv: https://arxiv.org/abs/2001.11782 · HTML: https://mlsanity.com/paper/2001.11782/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2020.

## Built on by (most cited first, 3 of 3)

- [Towards Adaptable and Interactive Image Captioning with Data Augmentation and Episodic Memory](https://mlsanity.com/paper/2306.03500/) (2023)
- [Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning](https://mlsanity.com/paper/2307.09915/) (2023)
- [Interactive Attention AI to translate low light photos to captions for night scene understanding in women safety](https://mlsanity.com/paper/2201.00969/) (2022)

## Abstract (arXiv)

In this paper we study a brand new topic of interactive image captioning with human in the loop. Different from automated image captioning where a given test image is the sole input in the inference stage, we have access to both the test image and a sequence of (incomplete) user-input sentences in the interactive scenario. We formulate the problem as Visually Conditioned Sentence Completion (VCSC). For VCSC, we propose asynchronous bidirectional decoding for image caption completion (ABD-Cap). With ABD-Cap as the core module, we build iCap, a web-based interactive image captioning system capable of predicting new text with respect to live input from a user. A number of experiments covering both automated evaluations and real user studies show the viability of our proposals.