# Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data

arXiv: https://arxiv.org/abs/1803.08314 · HTML: https://mlsanity.com/paper/1803.08314/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2018.

## Built on by (most cited first, 30 of 37)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Fine-grained Image Captioning with CLIP Reward](https://mlsanity.com/paper/2205.13115/) (2022)
- [D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2112.01551/) (2021)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions](https://mlsanity.com/paper/2305.17718/) (2023)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning](https://mlsanity.com/paper/2104.13872/) (2021)
- [Cross-Domain Image Captioning with Discriminative Finetuning](https://mlsanity.com/paper/2304.01662/) (2023)
- [Group-based Distinctive Image Captioning with Memory Attention](https://mlsanity.com/paper/2108.09151/) (2021)
- [DistinctAD: Distinctive Audio Description Generation in Contexts](https://mlsanity.com/paper/2411.18180/) (2024)
- [Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning](https://mlsanity.com/paper/2102.12550/) (2021)
- [Rethinking the Reference-based Distinctive Image Captioning](https://mlsanity.com/paper/2207.11118/) (2022)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [Do DALL-E and Flamingo Understand Each Other?](https://mlsanity.com/paper/2212.12249/) (2022)
- [Exploring Semantic Relationships for Unpaired Image Captioning](https://mlsanity.com/paper/2106.10658/) (2021)
- [Single-Modal Entropy based Active Learning for Visual Question Answering](https://mlsanity.com/paper/2110.10906/) (2021)
- [CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge](https://mlsanity.com/paper/2211.09371/) (2022)
- [No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning](https://mlsanity.com/paper/2409.03025/) (2024)
- [Learning to translate by learning to communicate](https://mlsanity.com/paper/2207.07025/) (2022)
- [Paraphrasing Is All You Need for Novel Object Captioning](https://mlsanity.com/paper/2209.12343/) (2022)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data](https://mlsanity.com/paper/2301.11174/) (2023)
- [Pragmatic Inference with a CLIP Listener for Contrastive Captioning](https://mlsanity.com/paper/2306.08818/) (2023)
- [Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation](https://mlsanity.com/paper/2306.13460/) (2023)
- [Improving Reference-based Distinctive Image Captioning with Contrastive Rewards](https://mlsanity.com/paper/2306.14259/) (2023)
- [Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation](https://mlsanity.com/paper/2409.15125/) (2024)
- [Emergence of Machine Language: Towards Symbolic Intelligence with Neural Networks](https://mlsanity.com/paper/2201.05489/) (2022)

## Abstract (arXiv)

The aim of image captioning is to generate captions by machine to describe image contents. Despite many efforts, generating discriminative captions for images remains non-trivial. Most traditional approaches imitate the language structure patterns, thus tend to fall into a stereotype of replicating frequent phrases or sentences and neglect unique aspects of each image. In this work, we propose an image captioning framework with a self-retrieval module as training guidance, which encourages generating discriminative captions. It brings unique advantages: (1) the self-retrieval guidance can act as a metric and an evaluator of caption discriminativeness to assure the quality of generated captions. (2) The correspondence between generated captions and images are naturally incorporated in the generation process without human annotations, and hence our approach could utilize a large amount of unlabeled images to boost captioning performance with no additional laborious annotations. We demonstrate the effectiveness of the proposed retrieval-guided method on COCO and Flickr30k captioning datasets, and show its superior captioning performance with more discriminative captions.