mlsanityData to 7 Oct 2026

MAFA: Managing False Negatives for Vision-Language Pre-training

arXiv 2312.06112 ↗First posted 11 Dec 2023cs.CV cs.AIcited by 9

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We consider a critical issue of false negatives in Vision-Language Pre-training (VLP), a challenge that arises from the inherent many-to-many correspondence of image-text pairs in large-scale web-crawled datasets. The presence of false negatives can impede achieving optimal performance and even lead to a significant performance drop. To address this challenge, we propose MAFA (MAnaging FAlse negatives), which consists of two pivotal components building upon the recently developed GRouped mIni-baTch sampling (GRIT) strategy: 1) an efficient connection mining process that identifies and converts false negatives into positives, and 2) label smoothing for the image-text contrastive (ITC) loss. Our comprehensive experiments verify the effectiveness of MAFA across multiple downstream tasks, emphasizing the crucial role of addressing false negatives in VLP, potentially even surpassing the importance of addressing false positives. In addition, the compatibility of MAFA with the recent BLIP-family model is also demonstrated. Code is available at https://github.com/jaeseokbyun/MAFA.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 73 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

60 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Improving Multimodal Datasets with Image Captioning19 Jul
  2. 2023BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization17 Jul
  3. 2023Too Large; Data Reduction for Vision-Language Pre-Training31 May
  4. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  6. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  8. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2019When Does Label Smoothing Help?cited by 985
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2021Robust fine-tuning of zero-shot modelscited by 785
  13. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  14. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  15. 2020Contrastive Learning with Hard Negative Samplescited by 619
  16. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  17. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  18. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  21. 2020Debiased Contrastive Learningcited by 354
  22. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  23. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  24. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  25. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  26. 2022Patching open-vocabulary models by interpolating weightscited by 187
  27. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  28. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
  29. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  30. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  31. 2019Polysemous Visual-Semantic Embedding for Cross-Modal Retrievalcited by 83
  32. 2021Incremental False Negative Detection for Contrastive Learningcited by 48
  33. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  34. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  35. 2020Regularization via Structural Label Smoothingcited by 29
  36. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  37. 2022NLIP: Noise-robust Language-Image Pre-trainingcited by 20
  38. 2022GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-trainingcited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 3.

9 papers.

  1. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  2. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  3. 2024Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracingcited by 11
  4. 2025CellCLIP -- Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learningcited by 6
  5. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4
  6. 2025GeoMM: On Geodesic Perspective for Multi-modal Learningcited by 2
  7. 2026Structure Observation Driven Image-Text Contrastive Learning for Computed Tomography Report Generationcited by 0
  8. 2026HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understandingcited by 0
  9. 2026Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment
  2. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models
  3. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  4. 2024Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples
  5. 2026What Makes Synthetic Hard Negatives Work in Vision-Language Pretraining?
  6. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images
  7. 2023RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models
  8. 2024POINTS: Improving Your Vision-language Model with Affordable Strategies