# Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner

arXiv: https://arxiv.org/abs/1705.00930 · HTML: https://mlsanity.com/paper/1705.00930/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2017.

## Built on by (most cited first, 21 of 21)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives](https://mlsanity.com/paper/2208.07422/) (2022)
- [M^4I: Multi-modal Models Membership Inference](https://mlsanity.com/paper/2209.06997/) (2022)
- [LabOR: Labeling Only if Required for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2108.05570/) (2021)
- [I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision](https://mlsanity.com/paper/2211.09778/) (2022)
- [Prompt-based Learning for Unpaired Image Captioning](https://mlsanity.com/paper/2205.13125/) (2022)
- [Cross-Domain Image Captioning with Discriminative Finetuning](https://mlsanity.com/paper/2304.01662/) (2023)
- [Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions](https://mlsanity.com/paper/2207.03133/) (2022)
- [Adma-GAN: Attribute-Driven Memory Augmented GANs for Text-to-Image Generation](https://mlsanity.com/paper/2209.14046/) (2022)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data](https://mlsanity.com/paper/2301.11174/) (2023)
- [VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks](https://mlsanity.com/paper/2407.19795/) (2024)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Improving Generalization of Image Captioning with Unsupervised Prompt Learning](https://mlsanity.com/paper/2308.02862/) (2023)
- [Learning Complex Users' Preferences for Recommender Systems](https://mlsanity.com/paper/2107.01529/) (2021)
- [REST: REtrieve & Self-Train for generative action recognition](https://mlsanity.com/paper/2209.15000/) (2022)
- [Guiding Attention in End-to-End Driving Models](https://mlsanity.com/paper/2405.00242/) (2024)
- [Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data](https://mlsanity.com/paper/2507.08610/) (2025)
- [Enhanced Modality Transition for Image Captioning](https://mlsanity.com/paper/2102.11526/) (2021)
- [Experimenting with Self-Supervision using Rotation Prediction for Image Captioning](https://mlsanity.com/paper/2107.13111/) (2021)
- [A Simple Baseline for Adversarial Domain Adaptation-based Unsupervised Flood Forecasting](https://mlsanity.com/paper/2206.08105/) (2022)

## Abstract (arXiv)

Impressive image captioning results are achieved in domains with plenty of training image and sentence pairs (e.g., MSCOCO). However, transferring to a target domain with significant domain shifts but no paired training data (referred to as cross-domain image captioning) remains largely unexplored. We propose a novel adversarial training procedure to leverage unpaired data in the target domain. Two critic networks are introduced to guide the captioner, namely domain critic and multi-modal critic. The domain critic assesses whether the generated sentences are indistinguishable from sentences in the target domain. The multi-modal critic assesses whether an image and its generated sentence are a valid pair. During training, the critics and captioner act as adversaries -- captioner aims to generate indistinguishable sentences, whereas critics aim at distinguishing them. The assessment improves the captioner through policy gradient updates. During inference, we further propose a novel critic-based planning method to select high-quality sentences without additional supervision (e.g., tags). To evaluate, we use MSCOCO as the source domain and four other datasets (CUB-200-2011, Oxford-102, TGIF, and Flickr30k) as the target domains. Our method consistently performs well on all datasets. In particular, on CUB-200-2011, we achieve 21.8% CIDEr-D improvement after adaptation. Utilizing critics during inference further gives another 4.5% boost.