# Intention Oriented Image Captions with Guiding Objects

arXiv: https://arxiv.org/abs/1811.07662 · HTML: https://mlsanity.com/paper/1811.07662/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2018.

## Built on by (most cited first, 20 of 20)

- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [Tag2Text: Guiding Vision-Language Model via Image Tagging](https://mlsanity.com/paper/2303.05657/) (2023)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Guiding Visual Question Generation](https://mlsanity.com/paper/2110.08226/) (2021)
- [Learning Combinatorial Prompts for Universal Controllable Image Captioning](https://mlsanity.com/paper/2303.06338/) (2023)
- [Semi-Autoregressive Image Captioning](https://mlsanity.com/paper/2110.05342/) (2021)
- [Question-controlled Text-aware Image Captioning](https://mlsanity.com/paper/2108.02059/) (2021)
- [Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation](https://mlsanity.com/paper/2210.11109/) (2022)
- [Interactive Machine Learning for Image Captioning](https://mlsanity.com/paper/2202.13623/) (2022)
- [Self-Annotated Training for Controllable Image Captioning](https://mlsanity.com/paper/2110.08446/) (2021)
- [CIC-BART-SSA: Controllable Image Captioning with Structured Semantic Augmentation](https://mlsanity.com/paper/2407.11393/) (2024)
- [Fine-grained length controllable video captioning with ordinal embeddings](https://mlsanity.com/paper/2408.15447/) (2024)
- [Macroscopic Control of Text Generation for Image Captioning](https://mlsanity.com/paper/2101.08000/) (2021)
- [Towards Subject-Oriented Video Captioning via User-Specified Targets](https://mlsanity.com/paper/2312.13330/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Caption Generation for Dongba Paintings via Prompt Learning and Semantic Fusion](https://mlsanity.com/paper/2603.22946/) (2026)
- [IntentQA: Intent Question Answering in Videos by Cognitive Context Reasoning](https://mlsanity.com/paper/2608.23330/) (2026)

## Abstract (arXiv)

Although existing image caption models can produce promising results using recurrent neural networks (RNNs), it is difficult to guarantee that an object we care about is contained in generated descriptions, for example in the case that the object is inconspicuous in the image. Problems become even harder when these objects did not appear in training stage. In this paper, we propose a novel approach for generating image captions with guiding objects (CGO). The CGO constrains the model to involve a human-concerned object when the object is in the image. CGO ensures that the object is in the generated description while maintaining fluency. Instead of generating the sequence from left to right, we start the description with a selected object and generate other parts of the sequence based on this object. To achieve this, we design a novel framework combining two LSTMs in opposite directions. We demonstrate the characteristics of our method on MSCOCO where we generate descriptions for each detected object in the images. With CGO, we can extend the ability of description to the objects being neglected in image caption labels and provide a set of more comprehensive and diverse descriptions for an image. CGO shows advantages when applied to the task of describing novel objects. We show experimental results on both MSCOCO and ImageNet datasets. Evaluations show that our method outperforms the state-of-the-art models in the task with average F1 75.8, leading to better descriptions in terms of both content accuracy and fluency.