mlsanityData to 7 Oct 2026

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

arXiv 2102.08981 ↗First posted 17 Feb 2021cs.CV cs.CLcited by 1,209

Not on the map. Why?

Cited by 387 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pre-training. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pre-training data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [Sharma et al. 2018] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for vision-and-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks.

Key facts. Cited by 1,209 papers in the map (data through 7 Oct 2026). Builds on 70 of its 100 references that are in the map. First posted on arXiv 17 Feb 2021.

Builds on

70 of 100 references.

References · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  13. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  14. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  15. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  16. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  17. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  18. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  19. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  20. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  21. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  22. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  23. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  24. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  25. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  26. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  27. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  28. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  31. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  32. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  33. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  34. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  35. 2019Attention on Attention for Image Captioningcited by 289
  36. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  37. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  38. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 152; 2023: 306; 2024: 332; 2025: 270; 2026: 118.

1,209 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
38 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  12. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  14. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  15. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  16. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  17. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  18. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  19. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  20. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  21. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  22. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  23. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  24. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  25. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  26. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  27. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  28. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  29. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  30. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  31. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  32. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  33. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  34. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  35. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  36. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  37. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  38. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434

50 most cited of 1,209 All 1,209 (JSON)