# Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach

arXiv: https://arxiv.org/abs/1909.02201 · HTML: https://mlsanity.com/paper/1909.02201/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 30 of 30)

- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [DASO: Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2106.05682/) (2021)
- [More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2103.13990/) (2021)
- [LabOR: Labeling Only if Required for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2108.05570/) (2021)
- [Prompt-based Learning for Unpaired Image Captioning](https://mlsanity.com/paper/2205.13125/) (2022)
- [A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision](https://mlsanity.com/paper/2112.13884/) (2021)
- [MCDAL: Maximum Classifier Discrepancy for Active Learning](https://mlsanity.com/paper/2107.11049/) (2021)
- [Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality](https://mlsanity.com/paper/2410.05210/) (2024)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillation](https://mlsanity.com/paper/2104.05965/) (2021)
- [Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning](https://mlsanity.com/paper/2104.13872/) (2021)
- [R3Net:Relation-embedded Representation Reconstruction Network for Change Captioning](https://mlsanity.com/paper/2110.10328/) (2021)
- [Generative Bias for Robust Visual Question Answering](https://mlsanity.com/paper/2208.00690/) (2022)
- [ACP++: Action Co-occurrence Priors for Human-Object Interaction Detection](https://mlsanity.com/paper/2109.04047/) (2021)
- [IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning](https://mlsanity.com/paper/2409.18046/) (2024)
- [Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition](https://mlsanity.com/paper/2211.00448/) (2022)
- [Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition](https://mlsanity.com/paper/2203.03195/) (2022)
- [Exploring Semantic Relationships for Unpaired Image Captioning](https://mlsanity.com/paper/2106.10658/) (2021)
- [Single-Modal Entropy based Active Learning for Visual Question Answering](https://mlsanity.com/paper/2110.10906/) (2021)
- [Multimodal Data Augmentation for Image Captioning using Diffusion Models](https://mlsanity.com/paper/2305.01855/) (2023)
- [Intra-agent speech permits zero-shot task acquisition](https://mlsanity.com/paper/2206.03139/) (2022)
- [Paraphrasing Is All You Need for Novel Object Captioning](https://mlsanity.com/paper/2209.12343/) (2022)
- [Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data](https://mlsanity.com/paper/2301.11174/) (2023)
- [ENInst: Enhancing Weakly-supervised Low-shot Instance Segmentation](https://mlsanity.com/paper/2302.09765/) (2023)
- [REST: REtrieve & Self-Train for generative action recognition](https://mlsanity.com/paper/2209.15000/) (2022)
- [The Solution for the CVPR2023 NICE Image Captioning Challenge](https://mlsanity.com/paper/2310.06879/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey](https://mlsanity.com/paper/2412.08158/) (2024)

## Abstract (arXiv)

Constructing an organized dataset comprised of a large number of images and several captions for each image is a laborious task, which requires vast human effort. On the other hand, collecting a large number of images and sentences separately may be immensely easier. In this paper, we develop a novel data-efficient semi-supervised framework for training an image captioning model. We leverage massive unpaired image and caption data by learning to associate them. To this end, our proposed semi-supervised learning method assigns pseudo-labels to unpaired samples via Generative Adversarial Networks to learn the joint distribution of image and caption. To evaluate, we construct scarcely-paired COCO dataset, a modified version of MS COCO caption dataset. The empirical results show the effectiveness of our method compared to several strong baselines, especially when the amount of the paired samples are scarce.