mlsanityData to 7 Oct 2026

Scaling Language-Image Pre-training via Masking

arXiv 2212.00794 ↗First posted 1 Dec 2022cs.CVcited by 317

Not on the map. Why?

Cited by 81 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Fast Language-Image Pre-training (FLIP), a simple and more efficient method for training CLIP. Our method randomly masks out and removes a large portion of image patches during training. Masking allows us to learn from more image-text pairs given the same wall-clock time and contrast more samples per iteration with similar memory footprint. It leads to a favorable trade-off between accuracy and training time. In our experiments on 400 million image-text pairs, FLIP improves both accuracy and speed over the no-masking baseline. On a large diversity of downstream tasks, FLIP dominantly outperforms the CLIP counterparts trained on the same data. Facilitated by the speedup, we explore the scaling behavior of increasing the model size, data size, or training length, and report encouraging results and comparisons. We hope that our work will foster future research on scaling vision-language learning.

Key facts. Cited by 317 papers in the map (data through 7 Oct 2026). Builds on 62 of its 75 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

62 of 75 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  13. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  14. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  16. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  18. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  19. 2016Deep Networks with Stochastic Depthcited by 973
  20. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  21. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  22. 2021Robust fine-tuning of zero-shot modelscited by 785
  23. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  24. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  25. 2017Large Batch Training of Convolutional Networkscited by 529
  26. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  27. 2018nocaps: novel object captioning at scalecited by 444
  28. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  29. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  30. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  31. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  32. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  33. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  34. 2022Masked Autoencoders that Listencited by 271
  35. 2022Masked Visual Pre-training for Motor Controlcited by 246
  36. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  37. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  38. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 96; 2024: 128; 2025: 67; 2026: 24.

317 papers.

Sort:

By area:

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  4. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  5. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  6. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  7. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  8. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  9. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  10. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  11. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  12. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
38 more
  1. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  2. 2023Improving CLIP Training with Language Rewritescited by 216
  3. 2023Glaze: Protecting Artists from Style Mimicry by Text-to-Image Modelscited by 200
  4. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  5. 2023Fast Training of Diffusion Models with Masked Transformerscited by 172
  6. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  7. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  8. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  9. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  10. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  11. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  12. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  13. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 123
  14. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  15. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  16. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  17. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  18. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  19. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  20. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  21. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  22. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  23. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  24. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  25. 2023Fine-Grained Visual Promptingcited by 81
  26. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  27. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  28. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  29. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  30. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  31. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  32. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  33. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 68
  34. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  35. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  36. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  37. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  38. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56

50 most cited of 317 All 317 (JSON)