Data to 7 Oct 2026

XGPT: Cross-modal Generative Pre-Training for Image Captioning

arXiv 2003.01473 ↗First posted 3 Mar 2020cs.CL cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

While many BERT-based cross-modal pre-trained models produce excellent results on downstream understanding tasks like image-text retrieval and VQA, they cannot be applied to generation tasks directly. In this paper, we propose XGPT, a new method of Cross-modal Generative Pre-Training for Image Captioning that is designed to pre-train text-to-image caption generators through three novel generation tasks, including Image-conditioned Masked Language Modeling (IMLM), Image-conditioned Denoising Autoencoding (IDA), and Text-conditioned Image Feature Generation (TIFG). As a result, the pre-trained XGPT can be fine-tuned without any task-specific architecture modifications to create state-of-the-art models for image captioning. Experiments show that XGPT obtains new state-of-the-art results on the benchmark datasets, including COCO Captions and Flickr30k Captions. We also use XGPT to generate new image captions as data augmentation for the image retrieval task and achieve significant improvement on all recall metrics.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 18; 2023: 15; 2024: 4; 2025: 4.

51 papers.

Sort:

By area:

2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
2022Multimodal Learning with Transformers: A Surveycited by 399
2021Pre-Trained Models: Past, Present and Futurecited by 319
2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
2022A Survey of Vision-Language Pre-Trained Modelscited by 177
2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
2022Pretrained Language Models for Text Generation: A Surveycited by 121
2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96

38 more

2021Pretrained Language Models for Text Generation: A Surveycited by 95
2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
2022Vision-and-Language Pretrained Models: A Surveycited by 50
2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Modelscited by 39
2023Transformers in Speech Processing: A Surveycited by 38
2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
2023Probabilistic Prompt Learning for Dense Predictioncited by 17
2022I-Tuning: Tuning Frozen Language Models with Image for Lightweight Image Captioningcited by 16
2021Iconographic Image Captioning for Artworkscited by 15
2022RepsNet: Combining Vision with Language for Automated Medical Reportscited by 14
2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12
2023A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directionscited by 12
2024AesopAgent: Agent-driven Evolutionary System on Story-to-Video Productioncited by 12
2022A survey on knowledge-enhanced multimodal learningcited by 11
2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
2021Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generationcited by 8
2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generationcited by 7
2022Every picture tells a story: Image-grounded controllable stylistic story generationcited by 5
2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
2022DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-trainingcited by 4
2022DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attentioncited by 4
2023A Survey on Image-text Multimodal Modelscited by 4
2024Retrieval Augmented Recipe Generationcited by 3
2022Vision-and-Language Pretrainingcited by 2
2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
2023i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Datacited by 2
2023Enhance Reasoning Ability of Visual-Language Models via Large Language Modelscited by 2
2024CIC-BART-SSA: Controllable Image Captioning with Structured Semantic Augmentationcited by 2
2023Multimodal Deep Learning for Scientific Imaging Interpretationcited by 1
2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Datacited by 1
2025Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMscited by 0

50 most cited of 51 All 51 (JSON)