# Fine-Grained Image Captioning with Global-Local Discriminative Objective

arXiv: https://arxiv.org/abs/2007.10662 · HTML: https://mlsanity.com/paper/2007.10662/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2020.

## Built on by (most cited first, 12 of 12)

- [FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation](https://mlsanity.com/paper/2303.17225/) (2023)
- [Prompt-based Learning for Unpaired Image Captioning](https://mlsanity.com/paper/2205.13125/) (2022)
- [Learning for Unconstrained Space-Time Video Super-Resolution](https://mlsanity.com/paper/2102.13011/) (2021)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [AlignDet: Aligning Pre-training and Fine-tuning in Object Detection](https://mlsanity.com/paper/2307.11077/) (2023)
- [Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition](https://mlsanity.com/paper/2203.03195/) (2022)
- [Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction](https://mlsanity.com/paper/2403.10883/) (2024)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [Image Captioning via Dynamic Path Customization](https://mlsanity.com/paper/2406.00334/) (2024)
- [Detailed Object Description with Controllable Dimensions](https://mlsanity.com/paper/2411.19106/) (2024)
- [Bypass Network for Semantics Driven Image Paragraph Captioning](https://mlsanity.com/paper/2206.10059/) (2022)
- [HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2604.16499/) (2026)

## Abstract (arXiv)

Significant progress has been made in recent years in image captioning, an active topic in the fields of vision and language. However, existing methods tend to yield overly general captions and consist of some of the most frequent words/phrases, resulting in inaccurate and indistinguishable descriptions (see Figure 1). This is primarily due to (i) the conservative characteristic of traditional training objectives that drives the model to generate correct but hardly discriminative captions for similar images and (ii) the uneven word distribution of the ground-truth captions, which encourages generating highly frequent words/phrases while suppressing the less frequent but more concrete ones. In this work, we propose a novel global-local discriminative objective that is formulated on top of a reference model to facilitate generating fine-grained descriptive captions. Specifically, from a global perspective, we design a novel global discriminative constraint that pulls the generated sentence to better discern the corresponding image from all others in the entire dataset. From the local perspective, a local discriminative constraint is proposed to increase attention such that it emphasizes the less frequent but more concrete words/phrases, thus facilitating the generation of captions that better describe the visual details of the given images. We evaluate the proposed method on the widely used MS-COCO dataset, where it outperforms the baseline methods by a sizable margin and achieves competitive performance over existing leading approaches. We also conduct self-retrieval experiments to demonstrate the discriminability of the proposed method.