mlsanityData to 7 Oct 2026

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

arXiv 2102.10407 ↗First posted 20 Feb 2021cs.CV cs.AI cs.CLcited by 190

Not on the map. Why?

Cited by 70 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model(LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder ona small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and upto 5.4% CIDEr on Conceptual Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset. To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data. Our code is available at: https://github.com/Vision-CAIR/VisualGPT.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Builds on 68 of its 98 references that are in the map. First posted on arXiv 20 Feb 2021.

Builds on

68 of 98 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2016Layer Normalizationcited by 5,579
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  13. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  14. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  15. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  16. 2018Object Hallucination in Image Captioningcited by 661
  17. 2016Self-critical Sequence Training for Image Captioningcited by 544
  18. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  21. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  22. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  23. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  24. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  25. 2019Attention on Attention for Image Captioningcited by 289
  26. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  27. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  28. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  29. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  30. 2018Exploring Visual Relationship for Image Captioningcited by 215
  31. 2020X-Linear Attention Networks for Image Captioningcited by 198
  32. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  33. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  34. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  35. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  36. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  37. 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
  38. 2018Neural Baby Talkcited by 107

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 18; 2023: 64; 2024: 56; 2025: 36; 2026: 14.

190 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  6. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  8. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  10. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  11. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  12. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
38 more
  1. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  2. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  3. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  4. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  5. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  6. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  7. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  9. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  10. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  11. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  12. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  13. 2023Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Modelscited by 104
  14. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  15. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  16. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  17. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 90
  18. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  19. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  20. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  21. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 85
  22. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  23. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 79
  24. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  25. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  26. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  27. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  28. 2023Learning to Model the World with Languagecited by 66
  29. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 60
  30. 2024ToolNet: Connecting Large Language Models with Massive Tools via Tool Graphcited by 59
  31. 2023When Brain-inspired AI Meets AGIcited by 54
  32. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Textscited by 51
  33. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answeringcited by 50
  34. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  35. 2022FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Contextcited by 47
  36. 2023Text-guided Foundation Model Adaptation for Pathological Image Classificationcited by 47
  37. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  38. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43

50 most cited of 190 All 190 (JSON)