# Contrastive Learning for Image Captioning

arXiv: https://arxiv.org/abs/1710.02534 · HTML: https://mlsanity.com/paper/1710.02534/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2017.

## Built on by (most cited first, 30 of 54)

- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Fine-grained Image Captioning with CLIP Reward](https://mlsanity.com/paper/2205.13115/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation](https://mlsanity.com/paper/2109.12242/) (2021)
- [Deep Matching Prior: Test-Time Optimization for Dense Correspondence](https://mlsanity.com/paper/2106.03090/) (2021)
- [Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection](https://mlsanity.com/paper/2207.05500/) (2022)
- [Prompt-based Learning for Unpaired Image Captioning](https://mlsanity.com/paper/2205.13125/) (2022)
- [Affection: Learning Affective Explanations for Real-World Visual Data](https://mlsanity.com/paper/2210.01946/) (2022)
- [Emergent Visual-Semantic Hierarchies in Image-Text Representations](https://mlsanity.com/paper/2407.08521/) (2024)
- [FCL-GAN: A Lightweight and Real-Time Baseline for Unsupervised Blind Image Deblurring](https://mlsanity.com/paper/2204.07820/) (2022)
- [CamDiff: Camouflage Image Augmentation via Diffusion Model](https://mlsanity.com/paper/2304.05469/) (2023)
- [DORA: Exploring Outlier Representations in Deep Neural Networks](https://mlsanity.com/paper/2206.04530/) (2022)
- [Cross-Domain Image Captioning with Discriminative Finetuning](https://mlsanity.com/paper/2304.01662/) (2023)
- [CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2106.10855/) (2021)
- [Group-based Distinctive Image Captioning with Memory Attention](https://mlsanity.com/paper/2108.09151/) (2021)
- [DistinctAD: Distinctive Audio Description Generation in Contexts](https://mlsanity.com/paper/2411.18180/) (2024)
- [Learning with Instance Bundles for Reading Comprehension](https://mlsanity.com/paper/2104.08735/) (2021)
- [Bilateral Personalized Dialogue Generation with Contrastive Learning](https://mlsanity.com/paper/2106.07857/) (2021)
- [Rethinking the Reference-based Distinctive Image Captioning](https://mlsanity.com/paper/2207.11118/) (2022)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [Image Captioning with Multi-Context Synthetic Data](https://mlsanity.com/paper/2305.18072/) (2023)
- [Efficient and Versatile Robust Fine-Tuning of Zero-shot Models](https://mlsanity.com/paper/2408.05749/) (2024)
- [FICGAN: Facial Identity Controllable GAN for De-identification](https://mlsanity.com/paper/2110.00740/) (2021)
- [Distinctive Image Captioning via CLIP Guided Group Optimization](https://mlsanity.com/paper/2208.04254/) (2022)
- [CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge](https://mlsanity.com/paper/2211.09371/) (2022)
- [COMICS: End-to-end Bi-grained Contrastive Learning for Multi-face Forgery Detection](https://mlsanity.com/paper/2308.01520/) (2023)
- [Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA](https://mlsanity.com/paper/2312.13594/) (2023)

## Abstract (arXiv)

Image captioning, a popular topic in computer vision, has achieved substantial progress in recent years. However, the distinctiveness of natural descriptions is often overlooked in previous work. It is closely related to the quality of captions, as distinctive captions are more likely to describe images with their unique aspects. In this work, we propose a new learning method, Contrastive Learning (CL), for image captioning. Specifically, via two constraints formulated on top of a reference model, the proposed method can encourage distinctiveness, while maintaining the overall quality of the generated captions. We tested our method on two challenging datasets, where it improves the baseline model by significant margins. We also showed in our studies that the proposed method is generic and can be used for models with various structures.