mlsanityData to 7 Oct 2026

FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models

arXiv 2405.10286 ↗First posted 16 May 2024cs.CV cs.AIcited by 7

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Despite noise and caption quality having been acknowledged as important factors impacting vision-language contrastive pre-training, in this paper, we show that the full potential of improving the training process by addressing such issues is yet to be realized. Specifically, we firstly study and analyze two issues affecting training: incorrect assignment of negative pairs, and low caption quality and diversity. Then, we devise effective solutions for addressing both problems, which essentially require training with multiple true positive pairs. Finally, we propose training with sigmoid loss to address such a requirement. We show very large gains over the current state-of-the-art for both image recognition (~ +6% on average over 11 datasets) and image retrieval (~ +19% on Flickr30k and ~ +15% on MSCOCO).

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 59 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

41 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Improved baselines for vision-language pre-training15 May
  2. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger30 Mar
  3. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention6 Mar
  4. 2022UniCLIP: Unified Framework for Contrastive Language-Image Pre-training27 Sep

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
25 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  5. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  6. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  7. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  8. 2020Are we done with ImageNet?cited by 329
  9. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  10. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  11. 2023Improving CLIP Training with Language Rewritescited by 216
  12. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  13. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  14. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  15. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  16. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  17. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  18. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  19. 2021Incremental False Negative Detection for Contrastive Learningcited by 48
  20. 2022Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positivescited by 37
  21. 2022Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervisioncited by 36
  22. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35
  23. 2022Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-trainingcited by 32
  24. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19
  25. 2022ProtoCLIP: Prototypical Contrastive Language Image Pretrainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  2. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  3. 2025Tables Guide Vision: Learning to See the Heart through Tabular Datacited by 3
  4. 2025Unlearning the Noisy Correspondence Makes CLIP More Robustcited by 1
  5. 2026REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Predictioncited by 1
  6. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretrainingcited by 0
  7. 2026UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
  2. 2021Contrastive Vision-Language Pre-training with Limited Resources
  3. 2023Image Captioners Are Scalable Vision Learners Too
  4. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
  5. 2026CLIMP: Contrastive Language-Image Mamba Pretraining
  6. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  7. 2023MAFA: Managing False Negatives for Vision-Language Pre-training
  8. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey