mlsanityData to 7 Oct 2026

Scaling Up Vision-Language Pre-training for Image Captioning

arXiv 2111.12233 ↗First posted 24 Nov 2021cs.CV cs.CLcited by 196

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In recent years, we have witnessed significant performance boost in the image captioning task based on vision-language pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art VinVL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million image-text pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail visual concepts when used in a zero-shot manner.

Key facts. Cited by 196 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 63 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

52 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  2. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
37 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2021Scaling Vision Transformerscited by 993
  5. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  6. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  11. 2018nocaps: novel object captioning at scalecited by 444
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  13. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  14. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  15. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  16. 2019Attention on Attention for Image Captioningcited by 289
  17. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  18. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  19. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  20. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  21. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  22. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  23. 2018Exploring Visual Relationship for Image Captioningcited by 215
  24. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  25. 2020X-Linear Attention Networks for Image Captioningcited by 198
  26. 2019Image Captioning: Transforming Objects into Wordscited by 171
  27. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  28. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  29. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  30. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  31. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  32. 2016Boosting Image Captioning with Attributescited by 76
  33. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  34. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  35. 2019Learning Video Representations using Contrastive Bidirectional Transformercited by 39
  36. 2020A Closer Look at the Robustness of Vision-and-Language Pre-trained Modelscited by 37
  37. 2020VIVO: Visual Vocabulary Pre-Training for Novel Object Captioningcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 47; 2023: 71; 2024: 42; 2025: 19; 2026: 16.

196 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  12. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
38 more
  1. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  2. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  3. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  4. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  5. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  6. 2022Multimodal Learning with Transformers: A Surveycited by 399
  7. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  8. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  9. 2022Learning Video Representations from Large Language Modelscited by 204
  10. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  11. 2022Translation between Molecules and Natural Languagecited by 173
  12. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  13. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  14. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  15. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  16. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  17. 2023Vision-Language Models as Success Detectorscited by 138
  18. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  19. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  20. 2023Large Language Models for Robotics: A Surveycited by 132
  21. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  22. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  23. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  24. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  25. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  26. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  27. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  28. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  29. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  30. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  31. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  32. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  33. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  34. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  35. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  36. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  37. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  38. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70

50 most cited of 196 All 196 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Too Large; Data Reduction for Vision-Language Pre-Training
  2. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation
  3. 2024DreamLIP: Language-Image Pre-training with Long Captions
  4. 2022VLP: A Survey on Vision-Language Pre-training
  5. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  6. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  7. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
  8. 2022A Survey of Vision-Language Pre-Trained Models